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Foreword 


The project-support role of the reliability assurance program makes it nec- 
essary to tailor each such program to the nature and requirements of the indi- 
vidual project. This situation creates a natural variation in the degree of so- 
phistication and proportioning of effort in each reliability program. In an effort 
to respond to this need for flexibility, there is frequently a tendency to lose sight 
of the primary need that each reliability program task, regardless of its level 
of sophistication, must contribute meaningfully to project decisions. This de- 
ficiency can be combatted effectively by astute management of the reliability 
assurance function, and one essential element in this management process— 
the subject of this publication— is the conduct of a continuous organized effort by 
the customer to monitor and evaluate the effectiveness of the reliability programs 
of the contractor and his subcontractors. 

Consistent with the objective of describing a project-wide and mission-wide 
perspective of the assurance effort, the degree of detail in discussion of method 
in this publication has been kept at a moderate level. 

The subject matter is presented here principally in the context of NASA proj - 
ect situations, particularly in chapter 4. However, the treatment of reliability 
program objectives, the criteria for judging task effectiveness, and the approach 
to evaluating the various tasks for meaningful accomplishment are essentially 
technical and are directed toward fundamentals. Thus, for the most part, this 
publication may be considered generic in its applicability. 

The authorship of this document has been a joint effort of D. S. Liberman of 
this office and A., J. Slechter of the ARINC Research Corporation. However, 
effort in constructively reviewing and commenting on the document has been ex- 
pended by numerous personnel of NASA program offices, NASA field installations, 
ARINC Research Corporation, and contractors to this office. This effort and the 
helpful suggestions offered by these reviewers are gratefully acknowledged. Spe- 
cial appreciation is expressed to the Jet Propulsion Laboratory of the California 
Institute of Technology for the use of its failure reporting system documents as 
supporting exhibits. 

It is expected that this publication will be of great usefulness as introductory 
material for NASA personnel new to the reliability assurance area. However, in 
view of its fundamental approach to the subject, it can serve a much wider use as 
a communication tool. 
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JohnE. Condon, Director 
Reliability and Quality Assurance 
Office of Industry Affairs 
NASA Headquarters 
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CHAPTER 1 

Introduction 


PURPOSE 

This document presents a basic orientation 
to the task of evaluating the effectiveness of are- 
liability program. Although evaluation method- 
ology is treated to some extent, primary em- 
phasis is devoted to discussing the assurance 
task as it relates to project requirements and 
resources and to describing the factors which 
determine effectiveness in program implemen- 
tation. The objective is to present an efficient 
general approach on which reliability assurance 
personnel can superimpose their own knowledge 
and initiative to adapt it for making timely and 
perceptive evaluations in their specific project 
situations. 

BACKGROUND 

Assuring the reliability of space -system 
hardware is a complex and difficult task. Bas- 
ically, reliability must be built into the hard- 
ware during its design and fabrication, and 
degradation of that reliability mustbe prevented 
in the succeeding steps leading to the hardware’s 
end use. One helpful technique in accomplishing 
this is the conduct of reliability and quality as- 


surance programs. Because of the high Levels 
of reliability required and the degree of com- 
plexity of space systems, policies and proce- 
dures have been developed to require contractors 
to plan and implement reliability and quality - 
assurance programs as a part of the design, 
fabrication, and utilization of major space - 
system hardware. 

Effective reliability and quality assurance 
programs require foresight, timely planning, 
and vigorous pursuit by both the customer and 
the contractor. In the reliability area, the pro- 
gram is based on a reliability program plan 
which must be implemented by the contractor 
and monitored by the customer. Generally sim- 
ilar procedures apply to quality assurance pro- 
grams, but detailed discussion of that area is 
outside the scope of this document. 

Monitoring is particularly important to the 
effective managementof the reliability program. 
Because the reliability effort is a support func- 
tion which must continually readjust to unfore- 
seen changes in the project program, continuous 
surveillance and assessment are necessary to 
assure that it does so effectively as the project 
progresses. 
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CHAPTER 2 


Foundations of Reliability Program 


purpose of reliability program 

Thebasic purpose of the reliability program 
is to contribute to the project in a supporting 
role, so as to increase the reliability of the 
hardware and systems produced. Often in the 
past, the contribution of the reliability program 
in this respect has been either small in com- 
parison with its cost or too difficult to identify, 
or the program has been too isolated. Hence, 
the attitude of both industry and government as 
to the value of reliability programs has often 
been a negative one. 

Unfortunately, reliability personnel in many 
cases have fallen short of taking the necessary 
positive action to assure the proper planning, 
execution, and management of their own pro- 
gram area. Further, because of the attitude 
which had resulted from weak past performance, 
even many sound recommendations from relia- 
bility personnel for improvements in their area 
often were not supported by management. 

This trend is now reversing, to a great ex- 
tent through a general recognition of the need 
for effective control disciplines to combat the 
failure potential resulting from increased mis- 
sion severity and increased hardware complex- 
ity. However, this improved attitude toward 
reliability programs cannot be expected to en- 
dure unless it is demonstrated in practice that 
the reliability program is truly effective in sup- 
plying this control discipline. 

As a basic rule in achieving its objective, 
it must be recognized that the reliability pro- 
gram performs its function effectively only as 
long as it supports the objectives and require- 
ments of the project. Functions beyond this ex- 
tent (unless specifically required by the contract 
for a stated secondary purpose) are academic 
and an unjustified use of resources. Therefore, 
reliability program management must take the 
initiative to ensure that the program is kept 
highly responsive to specific project needs, that 


it is staffed by adequate numbers of well- 
qualified personnel, and that it is conducted as 
an integral part of the project program. Oper- 
ated in this manner, the reliability program can 
offer an important contribution to project 
success. 

RESPONSE TO PROJECT CHARACTERISTICS 
General 

The principal project -situation factors that 
dictate differences in initial approach to the re- 
liability program are: 

(1) Mission criticality 

(2) System complexity 

(3) Type of mission 

(4) Relationship to state-of-the-art 

(5) Number of items produced 

(6) Proportion of hardware developed on 
subcontract 

(7) Use of existing hardware (including 
follow-ons) 

(8) Project status at inception of reliabil- 
ity program 

The following paragraphs discuss each of 
these factors as it affects the makeup of the 
reliability program. 

Mission Criticality 

Possibly the most basic indicator of mis- 
sion criticality is the cost of failure— that is, the 
ultimate cost (in terms of human life, dollars, 
and time) of loss of the mission which a given 
hardware item is required to perform. Fre- 
quently, the cost of the hardware itself is a good 
indicator of the magnitude of such loss . How- 
ever, in many cases (such as standard launch 
vehicles) unreliability of the hardware item of 
immediate concern can be responsible for far 
greater losses in the cost of payloads or even 
the costs of crippling delays in, or total loss of, 
a payload program. 
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The degree of criticality of a particular 
project will have a significant effect on shaping 
the efforts of the attendant reliability program. 
For the highly critical project, the reliability 
program must be conducted with greater thor- 
oughness, tighter discipline, and somewhat 
more formality in documentation than is re- 
quired for lesser projects. This emphasis 

would pervade almost all the reliability program 
areas and would dictate a need for increased 
manpower and greater dollar expenditures for 
reliability assurance. This is not to say that 
reliability program expenditures must be di- 
rectly proportional to the potential failure cost, 
but, where potential failure cost is high, there 
is ample justification for adequate funding of 
appropriately more thorough reliability efforts. 

System Complexity 

The general relationship between the order 
of system complexity and the risk of failure is 
well known. High orders of system complexity 
require much more extensive overall testing 
programs and place a heavy demand on the re- 
liability program to help minimize the effect of 
the greatly increased number of error sources 
(more components, more hardware interfaces, 
more organizational units, and more communi- 
cations links). The general effect will be to re- 
quire not only more technical effort in all re- 
liability task areas but also a larger and more 
demanding reliability program management 
effort . 

Type of Mission 

The type of mission to be performed by the 
hardware system is an important factor in de- 
termining the orientation and content of the re- 
liability program. A fairly broad breakdown of 
mission type may be made on the basis of the 
hardware classification, i.e., ground equip- 
ment, launch vehicles, or spacecraft. How- 
ever, in considering the effect on the reliability 
program, a more appropriate classification can 
be made based on a further consideration of dif- 
ferences in basic mission characteristics such 
as: 

(1) Manned or unmanned 

(2) Mission significance (importance of 
mission to the space program) 


(3) Mission life 

(4) Environmental stresses 

(5) Repair ability of hardware during mis- 
sion use 

(6) Extent and difficulty of data acquisition 
and tracking requirements 

(7) Extent of need for communication with, 
and command from, Earth 

It is difficult to make a single general statement 
on the overall effect of all these factors on a re- 
liability program. However, for any specific 
project, the appropriate areas for reliability 
program emphasis can be ascertained without 
great difficulty by considering these factors one 
by one . 

Relationship to State-of-the-Art 

Many NASA projects have certain mission 
obligations that approach the boundaries of the 
state-of-the-art either in design or in opera- 
tional techniques. Often, in fact, the advance- 
ment of the state-of-the-art is the primary rea- 
son for the project in that future program 
planning is dependent on the breakthrough. 

The effect of such conditions on the risk of 
failure places significant demands on the relia- 
bility program. Whereas reliance on proven 
techniques in design and fabrication is one of 
the basic methods by which high system relia- 
bility is achieved in the usual project situations, 
the project aimed at advancing the state-of-the- 
art must deliberately expose itself to risks in 
the area of the untried. Accordingly, in these 
cases, the reliability program must emphasize 
planned surveillance of the design, fabrication, 
and test cycle, with particular attention being 
devoted to the areas of design review, design- 
proof testing, and new component development 
associated with the advanced portions of the 
system. 

Number of Items Produced 

Whether a given project covers delivery of 
a large, as opposed to a small, number of cop- 
ies of the hardware item being developed can 
materially affect the shape of the reliability 
program required. The larger project will ex- 
perience a true production phase, with an op- 
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portunity for using more test items in the de- 
velopment cycle . In cases of this type (assuming 
equal orders of unit cost per item), the funding 
of a thorough reliability program can be more 
readily justified since the cost of the reliability 
effort can be amortized over a larger number of 
end items. 

On the other hand, the project which pro- 
duces only a few copies of the hardware item 
must evaluate the capability of the hardware 
with fewer test specimens; as a result, the re- 
liability program is more difficult to fund, but 
it is even more necessary in order to place all 
possible project emphasis on careful and 
thorough planning and analysis (throughout the 
design and preliminary development phases) to 
try to eliminate all possible failures on paper 
or in model testing prior to a commitment to 
flight-type hardware. 

Proportion of Hardware Developed on Subcontract 

Where a system procurement relies heavily 
on subcontracting, and particularly where a 
relatively large number of subcontractors are 
involved, the reliability program problems mul- 
tiply in much the same way that system manage- 
ment problems do. Effective reliability efforts 
by subcontractors are very important to overall 
mission success but are often difficult to ob- 
tain. Therefore the prime contractor's relia- 
bility program management effort must be par- 
ticularly strong, and in the technical reliability 
program areas care must be exercised to see 
that subcontractors use compatible method- 
ologies, that effective two-way communication 
is maintained to emphasize to the subcontractor 
the need for effective reliability effort, and that 
heavy emphasis is placed on those program 
areas which assure the adequacy of the "inter- 
facing” of the prime contractor's efforts (see 
also par. 2. 6 of NPC 250-1 (ref. 1)). 

Use of Existing Hardware 

Frequently, a system design will rely 
heavily on the incorporation of subsystems de- 
veloped under previous projects. The incorpo- 
ration of government furnished property (NPC 
250-1, par. 2.7) is one case of this type. In all 
such cases, the reliability program must em- 
phasize those areas which serve to ascertain 
the capabilities and limitations of the previously- 


developed elements of hardware ("shelf items") 
under the new conditions of use, so as to de- 
termine adequately what requirements for re- 
liability achievement must be placed on those 
elements which will involve new design work. 
As development proceeds, added emphasis must 
also be placed in the reliability evaluation area 
to assure that the shelf items are being inte- 
grated into the system without adversely af- 
fecting overall system reliability. 

A very special case of the use of existing 
hardware is the follow-on contract which em- 
bodies partial modification of an existing sys- 
tem to give additional mission capabilities or to 
improve its present operation. Since a relia- 
bility program will usually have been imple- 
mented previously for the original hardware, 
the reliability planning for the follow-on effort 
must be based on examining the previous re- 
liability program and reviewing the available 
reliability data and documentation to determine 
their applicability to the follow-on program. 
The new reliability program should then limit 
itself to the scope and depth necessary to sup- 
port the follow-on development. 

Project Status at Inception of Reliability Program 

There are cases in which extensions of con- 
tracts are negotiated while development is in 
midprocess. In these cases the existing relia- 
bility program often is a poor one, and one of 
the requirements to be imposed with the con- 
tract extension is the incorporation of a more 
effective reliability program. 

In such cases, particular care must be ex- 
ercised to assure that the extent to which each 
reliability task is invoked is appropriate for 
providing useful support to the project effort as 
of its current state of evolution and that the re- 
liability task cost will not be disproportionate 
to its expected impact on the project. 

Summary 

The foregoing factors roughly dictate the 
shape of the reliability program in that they de- 
scribe most of the basic project characteristics 
to which a reliability effort must respond. The 
more specific determination of the reliability 
program scope must be made in light of project 
management's decision as to the risks it is will- 
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ing to accept and the resources it is able to de- 
vote to reliability assurance after consideration 
of the project characteristics and of the eco- 
nomic constraints. 

Once the project is roughly categorized, 
pertinent economic factors should be taken into 
account and a determination made as to (1) where 
emphasis must be placed, (2) whether the com- 
plete list of reliability assurance tasks is ap- 
plicable, and (3) whether there are any ap- 
parently unique problems which must be 
overcome. This process of ascertaining project 
needs does not end with the establishment of a 
reliability program at the beginning of a project. 
It also applies to evaluation of changes in re- 
liability assurance needs after the project is 
well underway. 

NASA GUIDELINES AND PROCEDURES 

NASA has developed its policies and proce- 
dures in full recognition of the project support 
role of the reliability function and of the need 
for flexibility in designing reliability programs 
to fit a diversity of project needs. Two basic 
documents govern reliability programs for NASA 
procurements. One of these is NPC 400, (ref. 
2) "NASA Procurement Regulations," of which 
Part I, Subpart 51, is entitled "Integration of 
Reliability Requirements into NASA Procure- 
ments." The other document is NPC 250-1, 
(ref. 1), "Reliability Program Provisions for 
Space System Contractors." Essentially, NPC 
400 (ref. 2), part 1-51, prescribes the assign- 
ment of responsibilities within NASA for as- 
surance of reliability and gives procedures for 
establishing and managing reliability assurance 
efforts on contracts, and NPC 250-1 prescribes 
a generally applicable program of assurance 
tasks to be employed by contractors. These 
documents lay the procedural foundations for the 
reliability programs which are the concern of 
this text); therefore, a discussion of their es- 
sential features is considered important in 
putting the evaluation process on a sound foot- 
ing. Such a discussion is given in the following 
sections. 

Reliability Requirements in NASA Procurement Procedures 

The policy and outlook of NASA in regard to 
the area of reliability assurance are reflected 
in NPC 400 , 1-51 . Among the important features 


of that regulation are those which show the basic 
attitudes of the agency in this area and which 
form the pattern for the evaluation philosophies 
and approaches contained in this text. Some of 
these features are as follows: 

(1) Reliability assurance is defined as a 
systematic pattern of all actions necessary to 
provide adequate confidence that a space sys- 
tem, or portion thereof, will perform reliably 
in actual operation. The pattern of actions re- 
ferred to is further described in the definition 
by identification of appropriate actions in the 
various steps of the processes of program and 
project planning, procurement planning and ex- 
ecution, and use of the item. Among these are 
the actions to be taken in the planning and ex- 
ecution of contractor reliability programs, as 
well as in the government monitoring and guid- 
ance of such programs. 

(2) It is made clear that the intent of NASA 
policy as to the application of reliability pro- 
gram requirements (NPC 250-1, ref. 1) to proj- 
ects of various types is that the reliability pro- 
gram be tailored to the requirements and state 
of evolution of the project. 

(3) It is made clear that the primary re- 
sponsibility for the assurance of reliability for 
any project lies with the project manager, but 
it is intended that this responsibility be imple- 
mented with the advice and support of appro- 
priate reliability- assurance specialists. 

(4) NASA responsibility for defining clearly 
in procurement documents the extent of applica- 
bility of NPC 250-1 to a given procurement is 
stated, and a number of reliability program task 
areas where this is particularly necessary are 
identified . 

The reliability program evaluator must 
understand the logic and intent of the regulations 
fully and completely. This is essential to his 
making sound interpretations of that intent when 
judging a contractor's reliability program. 

Reliability Program Provisions for Contractors 

NASA reliability publication NPC 250-1 
furnishes a general guideline adaptable as a 
basic structure for reliability programs that fit 
project situations of various types. It presents 
the reliability effort as an organized program of 
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diverse tasks managed from a central point. 
The tasks themselves are functionally of two 
broad types: Those to be accomplished by re- 
liability personnel and those to be accomplished 
by personnel of other project areas (i. e., basic 
design disciplines) but monitored as a part of 
the reliability program. The common denomi- 
nator of all the tasks is their basic role in the 
planning and controlling of a sound design- 
development effort, wherein reliability status 
is periodically determined, timely inputs to de- 
sign trade-off decisions are provided, and engi- 
neering discipline is maintained over the funda- 
mental steps and milestones in the design and 
hardware evolution process. In Chapter 5 of 
the present text each reliability program ele- 
ment of NPC 250-1 will be treated in detail, and 
evaluation guidelines for each will be given. 
However, some of the key features and philoso- 
phies of that document are of particular interest 
here because they are fundamental to the evalu- 
ation effort; they are highlighted as follows: 

(1) It clarifies the relationship of the re- 
liability program to other project requirements 
and gives positive guidance for eliminating 
duplication of effort (par. 1.3). 

(2) It spells out the requirement for ac- 
cessibility to and visibility of all project data to 
NASA and its representatives, including inde- 
pendent reliability assessment contractors (par. 
1.4). Also, in order to provide for most con- 
venient accessibility, not only for NASA but also 
for the contractor’s own personnel, it pre- 
scribes the use of a central unified file (or data 
center) for all reliability and quality data (par. 
5.1). 

(3) It requires the conduct of the reliability 
effort as a formally organized program with 
central management, a documented program 


plan, and a separate accountability for relia- 
bility program resources (pars. 2.1, 2.2, and 
2.4). Moreover, it underscores the need for 
negotiating the reliability program together with 
the negotiation of the overall contract (rather 
than after contract execution). Also, it gives 
detailed procedures for this negotiation which 
provide for a realistic, three-step effort to 
delineate scope and cost of the reliability pro- 
gram and to develop the program plan (par. 2.2). 

(4) It requires periodic reviews of the re- 
liability program and provides for a revision of 
the Reliability Program Plan (RPP) to accom- 
modate changes found to be necessary as a re- 
sult of the reviews (par. 2.3). Since these re- 
views are jointly conducted by NASA and the 
contractor, they serve as one ideal means of 
implementing the recommendations of the re- 
liability program evaluation effort. 

(5) It requires that the prime contractor 
control the reliability of system elements ob- 
tained from subcontractors and that he de- 
termine the effect of reliability of system ele- 
ments provided as governmen t-furnished 
property on the reliability of the overall system 
(pars . 2.6 and 2.7). 

(6) It requires that the project be covered 
by one integrated test program instead of by 
separately managed testing programs in each of 
the various project areas (par. 4). This re- 
quirement helps prevent both duplications and 
omissions in testing and also provides a single 
test baseline, alongside of which the closely 
interrelated program of reliability assessment 
should be conducted. This approach empha- 
sizes the intimate tie-in of the reliability as- 
sessment effort with the requirements of the 
project and underscores its role as an input to 
the various project decision points. 


CHAPTER 3 


Keys to Effective Implementation 


The implementation of any task, whether in 
a reliability program or elsewhere, must satisfy 
certain basic criteria in order to be considered 
effective. The net effect of these criteria must 
be to determine whether the program, task, or 
other item serves a recognized need in a mean- 
ingful way. This section will discuss such ef- 
fectiveness criteria. These criteria may also 
be thought of as "evaluation tests" to which each 
element of the program should be subjected. 
Within the context of project needs, the evalua- 
tor should think in terms of the requirement 
that each program element must either satis- 
factorily pass each of these "evaluation tests" 
or undergo corrective action. 

Because of the complex and variable nature 
of the situations which a reliability program 
evaluator will face, it may be quite difficult (if 
indeed possible) to establish quantitative meas- 
urements of adequacy which are completely ob- 
jective. The judgment of the evaluator, there- 
fore, must be the dominant factor in determining 
whether the program element in question ade- 
quately satisfies the evaluation criteria. How- 
ever, if it is assumed that the evaluator is 
properly qualified, the absence of an objective 
quantitative yardstick does not impair the worth 
of the evaluation, since the very act of quantifi- 
cation in judgment-type situations is, in itself, 
highly subjective. The basic criteria for evalu- 
ation of any program task are: 

(1) Quality of performance 

(2) Timeliness 

(3) Cost in relation to need 

The effort put forth in each reliability pro- 
gram element must be of adequate quality, but 
not over-elaborate, for the actual project need; 
the output must be produced in time to have a 
positive effect on the project; and the amount of 
resources necessary to produce the output must 
be justified by its value to the project. Percep- 
tive application of these three tests to each ele- 


ment under evaluation would fairly assess 
whether the reliability program is efficiently 
providing a usable output to the project. 

However, the determination of effectiveness 
cannot end with a determination of the basic 
soundness of the planning and implementation of 
the reliability program tasks. It must also con- 
sider whether "usable" outputs are actually be- 
ing used. This question then dictates that two 
additional test criteria be applied to determine 
effectiveness; these are: 

(4) Attitudes of project personnel 

(5) Impact on project 

These last two factors, indicate the willing- 
ness and the ability of the project personnel to 
integrate effectively the reliability program out- 
puts into the design and development process. 

In the following sections these criteria are 
discussed in further detail. 

QUALITY OF PERFORMANCE 

Quality of performance is one of the pri- 
mary evaluation considerations. It can be 
simply stated: If the quality of the work within 
a program element is poor, there will be either 
little effect on the project or, worse, a degrad- 
ing effect. However, quality of performance 
cannot be meaningfully evaluated by itself. In 
considering "quality" in connection with a re- 
liability program task, it is necessary to be 
keenly aware that the objective of the task is to 
provide basic data, or otherwise to support a 
project decision, at some project decision point 
or milestone. When viewed in this light, the 
necessity of evaluating quality within the con- 
straints of timeliness and cost effectiveness 
becomes plainly apparent. 

It follows that the highest achievable quality 
of performance is not always justifiable. It is 
necessary that there be no compromise of valid- 
ity, competence, or accuracy (the term "com- 
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promise" refers to accepting a standard below 
a technically responsible level). However, 
careful judgment must be applied to avoid un- 
necessary or unwarranted levels of precision, 
elaborateness, refinement, orthoroughness. 

The best general approach to evaluating 
quality of performance might be to consider first 
the particular project decision point for which a 
given reliability task output is to be used. Nor- 
mally, this will give an insight into the order of 
elaborateness, accuracy, and precision re- 
quired. Next, the evaluator should consider the 
practical ease (and cost) of attaining the re- 
quired level of quality. Lastly, he should con- 
sider whether an adequate reliability task output 
can be reasonably produced within these con- 
straints and whether the performance he ob- 
serves is adequate within this frame of refer- 
ence. 

TIMELINESS 

Timeliness is perhaps the single most im- 
portant factor in differentiating between effective 
and ineffective implementation of a reliability 
program task. Since the purpose of the relia- 
bility program is to support the project, it is 
essential that the reliability program tasks be 
conducted integrally with the other project tasks 
and provide usable inputs at project decision 
points . Otherwise , the reliability input , whether 
it be analytical or of a monitoring nature, de- 
generates to an academic function or a histori- 
cal documentation process, instead of actively 
contributing to the improvement of hardware 
reliability. 

In evaluating timeliness of a reliability pro- 
gram task, it is essential to consider the nature 
of the design and development process itself. In 
essence, this process is an iterative one. It 
begins with conceptual studies which form the 
basis for selecting the overall design approach. 
This, in turn, provides the general point of de- 
parture for formulating concepts and designing 
subsystems and components. As the design and 
development process proceeds, the definition of 
the subsystems and components that will per- 
form the system’s mission functions becomes 
progressively clearer. At intervals throughout 
this process, engineering assessments must be 
made of the effect of one subsystem or com- 
ponent on the others or of the need to cope with 
technical problems not previously anticipated. 


This is particularly true in such areas as de- 
mands on the power or weight budgets or on 
limitations in attainable performance of one 
element, which place the burden for achieve- 
ment of overall performance goals on other ele- 
ments of the system. In addition, management 
assessments must be made where cost, sched- 
ule, and performance factors were examined. 

The purpose of all these assessments is to 
determine where and how the course of the 
project must be corrected to provide the opti- 
mum chance (both technical and managerial) of 
attaining overall project goals — in short, to 
guide project-directing decisions. These as- 
sessments must be made at all major mile- 
stones, and a formal design review is usually a 
part of them. This type of activity is not con- 
fined to the system level; it must also occur at 
the subsystem and component level, where the 
same iterative assessment/decision process 
takes place to conform with the design and de- 
velopment milestones that occur at the lower 
levels of assembly. 

The objective of the tasks in the contractor 
reliability program is to support the decision- 
making process. Analytical tasks provide a 
picture of relative (or estimated) reliability 
status, or they identify or quantify sources of 
potential failure. On the other hand, the moni- 
toring tasks- assure the decision maker that the 
progress accomplished at that point in time is 
soundly based and within the accepted bounds of 
the technical discipline. If the reliability pro- 
gram task (for either of these types) fails to 
provide usable information for the decision 
maker at or before the decision point, then it 
has made no contribution to that project de- 
cision. 

Obviously, it is not practical for every re- 
liability task to make a strong contribution to 
every project decision point. Some must be 
planned to impact only selected decision points. 
In either case, the output of the reliability task 
may sometimes slip its schedule and thus not be 
available at the decision point for which it was 
originally planned. Whenever a reliability task 
effort falls behind in this manner, the decision 
on whether to continue it will depend , in each 
particular case, on the probability that the task 
can "catch up" and contribute at a subsequent 
project decision point. 


KEYS TO EFFECTIVE IMPLEMENTATION 


11 


COST IN RELATION TO NEED 

A trade-off between the cost and the com- 
prehensiveness of a task is mandatory in almost 
every program situation. Reliability program 
functions are certainly no exception. In fact, 
because these functions may frequently be con- 
sidered a "soft spot” for cost trimming, the re- 
liability program managers must be constantly 
alert and assume the initiative in maintaining 
the cost-effectiveness aspect of all parts of their 
own program. 

In general, the evaluation must diligently 
pursue all reliability- assurance tasks to de- 
termine whether the contractor is using "gold- 
plated 1 ’ approaches in areas where only limited 
effort is required. The interrelated factors of 
quality and timeliness , covered in the preceding 
paragraphs, bear heavily on this. However, in 
addition to these, the appropriateness of the 
current scale of task implementation— or, 
simply, "currentness"— deserves particular 
mention in the area of cost management. The 
evaluator must bear in mind that the level of ef- 
fort assigned to a given task area at the time of 
initial reliability program planning is a "best 
estimate" as of the time at which it was made. 
Realism demands that these estimated levels be 
reexamined at the time of the reliability pro- 
gram evaluation to determine whether they have 
responded to the fluctuations in need dictated by 
the exigencies of the project. Frequently, 
originally planned levels of effort may later 
prove to be too "comfortable" in light of a re- 
duced need later in the project (although some- 
times the reverse may be just as true). There- 
fore, each time the reliability program is 
evaluated, the evaluator must review his earlier 
judgment of whether the scale on which each re- 
liability program task is implemented is still 
appropriate to the potential benefit of the task 
output to the project. These judgments should 
be used as a prime criterion for evaluating the 
efficiency of implementation of each reliability 
task, as well as that of the overall reliability 
program. 

ATTITUDES OF PROJECT PERSONNEL 

The success of the reliability program in 
supporting the design and development process 
is contingent upon effective working relation- 


ships between the reliability personnel and all 
other project personnel. An important part of 
this is a mutual agreement from the start as to 
what is expected of the reliability program. Al- 
though the careful planning of reliability tasks 
to provide timely and useful outputs to the proj- 
ect provides the foundation for a successful re- 
liability program, it is vital that the various 
user groups in engineering and manufacturing 
recognize and accept that a worthwhile support 
function is being competently performed by 
qualified personnel. 

As previously discussed in the section en- 
titled "Purpose of Reliability Program," there 
has been a negative attitude by project person- 
nel in many quarters toward reliability inputs; 
this attitude has largely resulted from the fail- 
ure of many past reliability programs to make 
effective or efficient project contribution. It is 
not reasonable to expect in-line project per- 
sonnel to respect a support function which not 
only appears to generate "useless" information 
but which also seems, in their eyes, to impede 
project progress. Therefore, it is extremely 
important that the reliability personnel strive 
not only to produce a worthwhile output in an ef- 
ficient manner but also to demonstrate through 
their daily performance that the reliability tasks 
are of material use to the project in an integral 
support role. 

Although it is the responsibility of relia- 
bility personnel to earn and maintain respect 
through good performance, company manage- 
ment must accept the basic responsibility for, 
and take the initiative in, developing positive 
attitudes toward the reliability function. Man- 
agement must ensure strong staffing (in quality, 
if not in numbers) of the reliability function and 
give it appropriate organizational stature. It 
should also see that the reliability function (both 
analytical and monitoring tasks) is performed 
and recognized as an integral part of the design 
and development process. When such support is 
given by management , and when this support is 
augmented by appropriate training of the engi- 
neering staff in the general application of reli- 
ability techniques, the necessary understanding 
and attitudes can be developed to promote ef- 
fective program implementation. 

The item of attitude of nonreliability per- 
sonnel will more frequently be encountered on a 
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program-wide or on an individual-person basis 
rather than task by task. However, the evalua- 
tor should be alert to this item on any basis, 
since it will provide important insight for his 
overall task. 

IMPACT ON PROJECT 

The final and most important test of the ef- 
fectiveness of any reliability program task is its 
impact on the project. Even if a task output is 
of adequate quality , is timely, is done efficiently, 
and is respected by in-line project personnel, it 
is not effective unless it has a constructive ef- 
fect on the project. Such effects might be cate- 
gorized as (1) direct support to improvement of 
hardware reliability, (2) providing confidence in 
the probability of achieving project success, or 
(3) indirect contribution to project success 
through increasing the consciousness of the 
need for good project discipline and in motiva- 
tion of personnel. 


Although motivational contributions are 
very hard to identify and evaluate , the functions 
of providing direct support or of providing con- 
fidence are evidenced in the use of reliability 
program outputs in the making of project de- 
cisions. In the area of direct support, the con- 
tributions might concern detection of points of 
weakness or sources of unreliability; or, more 
frequently, they might provide guidance as to 
the relative magnitude of risks from more than 
one potential failure source to help support a 
trade-off decision. The evaluator’s task of try- 
ing to detect the contribution of the reliability 
program to project decisions is a difficult one 
indeed. Although some direct indication can be 
obtained by observing the design review pro- 
gram, most of the bases for evaluation are in- 
direct, and some require a subjective judgment 
based on candid indications of the true attitude 
of both working level and key in-line project 
personnel. 


CHAPTER 4 

Evaluation Process 


AIMS AND REQUIREMENTS 

The previous sections have discussed in 
depth the philosophy behind employing reliability 
programs and have specified the general criteria 
governing the establishment and effective imple- 
mentation of such programs. With this founda- 
tion, one can now begin responding in detail to 
such questions as : 

(1) How is a reliability program evaluation 
best accomplished? 

(2) What type of personnel and resources 
are required, and where are they best deployed ? 

(3) What is the most effective organiza- 
tional approach to the evaluation problem ? 

(4) How are reliability program weaknesses 
detected and corrected? 

In providing the guidelines for answering 
these and other queries , one must first answer 
the more basic question: "What is the purpose 
behind the monitoring and evaluation effort?" 

Few would question the appropriateness of 
monitoring a contractor^ efforts as a general 
principle of contract management, and many 
would quickly equate monitoring to assessing 
the adequacy of compliance with contractual re- 
quirements. However, the problem is, "How 
does one define adequate compliance?" Despite 
the fact that the reliability program tasks are 
defined in detail in the reliability program plan 
(RPP) , the complexity of many of these tasks 
makes it extremely difficult to describe each 
element comprising each task in sufficient depth 
to define clearly what constitutes adequacy (or 
inadequacy). As a result, the evaluation which 
is oriented to compliance alone is likely to be 
ineffective in most project situations. 

A preferable approach is that of using the 
"compliance" criterion as one of the broader 
aspects of the evaluation, but to make the pri- 
mary aim of the evaluation a search for those 
areas of the reliability program which require 


improvement; the purpose is to discover the 
weak areas in sufficient lime to make the needed 
improvements (or eliminate activities which are 
wasted effort) . This objective , the prompt cor - 
rection of inadequacy, is the only immediately 
constructive reason for determining that an area 
of noncompliance exists (even if noncompliance 
can be clearly established). The other reason, 
assessment of penalties, normally comes well 
after the fact and, in any case, offers cold 
comfort to those whose primary interest is to 
prevent mission failures. 

The evaluation effort should be particularly 
concerned with determining: 

(1) Adequacy of the reliability program in 
meeting current project needs 

(2) Effectiveness of reliability program 
implementation in terms of input to, and impact 
on, the overall program . 

This approach emphasizes the need for re- 
liability program flexibility in keeping abreast 
of important changes in the project. Its prin- 
ciple, furthermore, is recognized in reliability 
publication NPC 250-1 (ref. 1) (par. 2.3), which 
requires periodic review of the reliability pro- 
gram to determine the need for, and to make 
appropriate changes to, that program. Finally, 
this approach is designed to help identify inef- 
fective reliability program elements that offer 
only an external appearance of adequacy, since 
it recognizes "compliance" in terms of effective 
compliance. 

Obviously, the evaluation effort cannot and 
should not assay to achieve a "one-for-one" 
coverage on all technical output under the con- 
tractor^ reliability program. Instead, its 
objective should be to selectively sample and 
evaluate enough of the various technical items 
to ascertain whether each is reasonably satis- 
factory for its purpose and whether the con- 
tractor^ reliability management group is doing 
an adequate job. 
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ORGANIZATION OF EVALUATION EFFORT 

To be effective , reliability program evalu- 
ation should be conducted as an orderly process. 
It should be managed in a positive, planned, and 
scheduled manner that effectively uses readily 
available resources and promptly identifies for 
correction those problems encountered in the 
evaluation process itself. 

As a first step in planning the evaluation 
effort, a careful review should be made of all 
the elements of the contract reliability program 
and the inputs to, and outputs from, each. These 
inputs and outputs should then be characterized 
as to form, degree of formality, ease of acces- 
sibility, physical location, milestone "time,” 
and, most important, the effort and degree of 
technical and management competence required 
to evaluate them intelligently. 

The next basic step should be a review and 
careful evaluation of the resources available to 
conduct the evaluation process. A complete 
listing of potential resources is given below, 
although there are many programs in which 
available evaluation help is far more limited. 

(1) Personnel of the NASA center itself 

(2) NASA representatives resident in the 
contractor’s plant 

(3) Resident in-plant representatives of 
other government agencies 

(4) Periodic visit surveys by a task group 

(5) Use of independent reliability assess- 
ment contractors 

Few programs exist in which any one of the 
above-listed resources can effectively satisfy 
the complete evaluation function. However, 
each has qualifications and location advantages 
which enable it to perform best certain portions 
of the total task. Note that each of these re- 
sources can vary widely from one project situa- 
tion to another in such respects as motivation, 
availability of manpower, levels of capability, 
and distribution of technical specialties. These 
attributes must be carefully appraised for the 
particular situation at hand during the planning 
of the reliability evaluation program. 

After fully assessing the job to be done and 
the means available to do it, the manager of the 
reliability program evaluation effort can then 


divide the overall task into parcels appropriate 
for taking maximum advantage of the available 
evaluation resources . 

The remainder of this chapter will further 
discuss the management and execution of the 
evaluation and monitoring program by describ- 
ing the role of the NASA center, the functions 
which can be served by various monitoring 
delegates, and the basic task-evaluation tech- 
nique. 

THE NASA CENTER 

The cognizant NASA center must be the 
focal point of the reliability program evaluation 
process. It is the project reliability assurance 
function at the NASA center that is responsible 
for general direction and motivation of the sys- 
tem contractor in the reliability area. This 
function must also interpret reliability policy 
and provide program guidelines to definitize 
general NASA requirements to meet the specific 
needs of the project. Included in this responsi- 
bility is the requirement for monitoring and 
evaluating the contractor’s reliability program. 

As brought out in the preceding section, the 
evaluation and monitoring of the contractor’s 
reliability program should be organized and 
systematic — in effect, a program in itself. The 
project reliability assurance function at the 
NASA center must accept the responsibility for 
planning and managing this evaluation program. 
Normally, those responsible for this function 
will also participate heavily in the technical 
evaluation work itself. These activities are 
discussed in the following sections. 

Planning and Management 

Initial Planning 

Planning the evaluation and monitoring ef- 
fort is a challenging task. Not only must one 
face the normal array of uncertainties that most 
planning functions encounter, but one must also 
deal with the problem of deciding what work can 
be delegated and who of the potential delegates 
can and will accomplish it properly. In addition 
to these difficulties, the demanding job of plan- 
ning the monitoring and evaluation effort occurs 
during the same time frame as other basic re- 
liability assurance tasks. Among these are the 
negotiation of the reliability program and the 
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initial shaping of the formal RPP. The NASA 
reliability assurance personnel should resist 
any temptation to classify mentally the moni- 
toring and evaluation effort as something of 
secondary importance or something which can 
be "taken care of later." The basic elements 
of planning the evaluation effort tie in very 
closely with the detailed development of the 
contractor’s RPP, which NASA must approve. 
In particular , the selected scheme of monitoring 
and evaluation should have a significant bearing 
on the documentation requirements under the 
RPP. Also, the point of acceptance or "Review” 
of certain contractor actions can sometimes be 
constructively delegated if the resident NASA 
representative's office is adequately staffed and 
willing to accept these responsibilities. Fur- 
ther, the decision on the need for contractual 
assistance in the evaluation effort should be 
made* as early as possible to enable the study 
contract effort to be initiated early in the hard- 
ware contract cycle. 

Apportionment of Effort 

In general, all monitoring and evaluation 
effort must be either delegated, contracted (as 
supporting studies), or performed by the relia- 
bility assurance personnel of the cognizant cen- 
ter itself. It is vital that the center be as 
realistic as possible in dividing this effort so 
that the program will be one of active and use- 
ful evaluation and will not degenerate into unin- 
formed and uninspired "onlooking" and "sight- 
seeing. " 

The proper sequence of decision in the ap- 
portionment process should normally be as 
follows: 

(1) What potential government-type dele- 
gates are available, and which tasks they can be 
assigned. 

(2) Of the remaining evaluation tasks , what 
can be handled by reliability personnel at the 
NASA center (including temporary duty at the 
contractor’s plant). 

(3) What complementary areas can be 
covered reasonably by visiting "survey" groups. 

(4) What is not adequately covered. Areas 
in this category can sometimes be handled ef- 
fectively by the use of the reliability study con- 


tract. These contracts are discussed under 
that heading (p. 21). 

Technical Participation 

The basic technical task to be accomplished 
by reliability assurance personnel at the NASA 
center is the overall technical supervision of the 
evaluation process. This task includes integra- 
tion of the inputs from all the various monitor- 
ing sources so that a composite evaluation pic- 
ture of the contractor’s reliability program can 
be built. This function may be broken down 
broadly into the following areas: 

(1) Evaluating report inputs 

(2) Planning and conducting periodic sur- 
veys 

(3) Special on-site evaluations 

(4) Overall evaluation and followup 

Evaluating Report Inputs 

The report inputs that the NASA center 
evaluator must be concerned with will be of 
two kinds: (1) The technical and management 
reports generated by the contractor as an out- 
put of the reliability program itself and (2) the 
monitoring reports received from the various 
monitoring and evaluation delegates. The eval- 
uation of these reports will serve to provide a 
picture of the effectiveness of both the evaluation 
effort and the contractor’s reliability program. 
Each report should be evaluated both as a sepa- 
rate information item and with a view toward 
identifying inconsistencies in the overall picture 
due to the failure of seemingly adequate indi- 
vidual items to fit together properly. Thus the 
evaluation of these reports not only can reveal 
the degree of adequacy of performance but also 
can guide the apportionment of further evalua- 
tion effort to determine causes of identified 
weakness, to clarify doubtful areas, and to 
resolve areas in which conflicting data are re- 
ceived from different sources. In particular, 
the information obtained from evaluation of re- 
port inputs can be most valuable in the planning 
of surveys . 

Normally, monitoring reports will be sub- 
mitted to the NASA center by the delegates as a 
task of their overall evaluation routine. How- 
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ever, the extent to which technical reports will 
be submitted depends on the terms of the con- 
tract. Where these are not required to be sub- 
mitted by the contractor, the evaluator must 
take the initiative to obtain them . He should 
sample the technical reports in the various re- 
liability task areas throughout the program to 
the extent he finds necessary to maintain ade- 
quate surveillance. 

The evaluation of individual technical re- 
port inputs obviously requires technical com- 
petence in the specialty areas being reported. 
However, this by itself is not enough, since 
quality and degree of sophistication of the re- 
port should depend on its intended use. There- 
fore, the evaluator must also have a firm ap- 
preciation of the use context of the particular 
item he is evaluating. 

Thus, the evaluation of report inputs can 
provide indications of many aspects of the con- 
tractor’s reliability program and of the ef- 
fectiveness of various aspects of the evaluation 
effort itself. However, it must be reemphasized 
that the obtaining of a really good "reading in 
depth" demands a broad competence and astute- 
ness on the part of the evaluator. 

Periodic Surveys 

As used here, the term "survey” implies 
an orderly and organized evaluation visit to a 
contractor’s facility by a task group of customer 
representatives. The periodic survey of the 
contractor’s reliability program serves several 
important functions in the overall evaluation 
process. 

By utilizing a formal and perceptive ap- 
proach, the survey helps to stress to the con- 
tractor the customer’s vital interest in a well- 
managed reliability assurance program. For 
the evaluator, it can be a means to obtain a 
complete overview and broad indication of the 
efficiency of the reliability program in a short 
period of time. However, most importantly, 
the survey performs the function of comple- 
menting the other facets of the evaluation pro- 
gram by directing specific attention to the reso- 
lution of questionable items identified through 
those other program facets and by obtaining a 
nonresident viewpoint. 

Generally speaking, the survey should be 
used in the overall evaluation scheme on a fairly 


regular schedule. However, the frequency will 
vary with a number of factors, the most im- 
portant of which are need, as evidenced by 
events in the contract situation at hand, and the 
resources of time and manpower available from 
the center to conduct the surveys . 

It is not within the scope of this text to treat 
the conduct of surveys in intimate detail. There 
are other sources, both within NASA and out- 
side, which provide specific instruction on the 
conduct of reliability program surveys. How- 
ever, it is appropriate here to cite some of the 
fundamentals for planning and conduct of the 
survey. 

The survey contributes most meaningfully 
to the overall evaluation program when it is 
planned as a complement to other monitoring 
and evaluation activities . This is far more ef- 
fective than using it as an independent evalua- 
tion mechanism. The formulation of survey 
objectives should, therefore, be based on indi- 
cations received from these other evaluation 
sources (technical and management reports both 
from delegates and the contractor; see preceding 
section). Normally, unless this is done, the 
survey either will end up trying to exceed its 
resources or will otherwise spend its effort un- 
productively;the net effect will be to accomplish 
little more than a show of interest to the con- 
tractor. Survey results that are vague and 
cursory will have little value in formulating a 
meaningful overall reliability program evalua- 
tion. 

The proper planning and conduct of a survey 
will include the following: 

(1) Formulation of objectives and estab- 
lishment of scope 

(2) Selection of a team 

(3) Advanced team preparation 

(4) Conduct of survey visit 

(5) Utilization of findings 

These elements are discussed in the suc- 
ceeding paragraphs . 

(1) Objectives and Scope. —The objectives 
and scope of the survey may vary widely, de- 
pending on the overall project situation in ques- 
tion. However, if a balanced evaluation effort 
is assumed, one of the most self-defeating 
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errors a survey can make is to try to cover a 
broader scope of investigation than its resources 
(time and manpower) can support. Careful con- 
sideration should be devoted to selection of ob- 
jectives. In so doing, it is perhaps better to 
plan to limit the scope of investigation to permit 
the vital areas of current question to be covered 
thoroughly at the expense of somedesirable-but- 
not-essential areas than to plan a scope of effort 
that will cause the team to be pressed for time . 
In the former case, at least the immediate ob- 
jectives of the survey can be assured, while, in 
the latter, the result may well be a "diluted" 
coverage which fails to achieve the desired in- 
sight into the essential problem areas. 

If available resources will not give even 
adequate coverage to all areas considered es- 
sential, a trade-off decision must be made 
wherein some of the less vital of these "es- 
sential" areas should be deferred for a future 
survey. The benefits to be derived in being 
thorough in the most vital areas will usually 
make this course worthwhile. On the contrary, 
a survey effort that is "spread too thin" will 
often fail to identify both the obvious program 
weaknesses and those that the contractor may 
be attempting to gloss over. 

(2) Survey Team Selections . —The manning 
of the reliability program survey team will vary 
with a number of considerations, the limiting 
factor in most cases being the availability of 
required personnel. This constraint will place 
a natural ceiling on both survey team size and 
practically achievable survey frequency. How- 
ever, the limitation of team size, by itself, is 
not particularly detrimental if the personnel 
used are of the proper levels of competence. 
This competence not only must cover the par- 
ticular technical specialties required, but the 
members must also be well grounded in the 
fundamentals of the research and development 
(R&D) process and astute in recognizing appli- 
cation of these fundamentals in the day-to-day 
workings of an industrial organization engaged 
in sizable R&D aerospace projects. 

This background is essential for enabling 
the surveyor to know where to look for objective 
evidence of the manner of accomplishment of a 
program task and is the foundation for the in- 
sight necessary to recognize the true signs of 
an effective or ineffective effort. It follows, 


therefore, that in the selection of personnel for 
the survey team greatest emphasis should be 
placed on this overall competence rather than 
on the job title of the individual’s position. As- 
suming that appropriate basic technical spe- 
cialists are not left out, the composition of the 
team may include personnel from quality, test, 
project engineering, etc. in addition to relia- 
bility assurance personnel. 

(3) Survey Preparation. —In order to pro- 
vide for fully productive use of the time spent by 
the survey team in the contractor's facility, it 
is essential that a thorough job of preparation be 
accomplished before the team sets out. The in- 
dividual team members should have a firm 
grasp of all pertinent background information 
including the following: 

(a) Contract requirements for relia- 
bility and quality assurance (R&QA) 

(b) Pertinent task requirements in the 
original and the current RPP 

(c) The manning of the tasks and the 
organizational interrelationships of personnel 
responsible for their accomplishment 

(d) The content and status, as practi- 
cable, of the contractor’s technical reports on 
the particular task areas in question 

(e) The content of reports by resident 
monitoring personnel on the task areas in ques- 
tion 

(f) Pertinent project factors or de- 
velopments (funding , schedules , priorities , 
special problems, etc.) which result from or 
bear on the reliability program. 

Each member should then chart his own survey 
plan to obtain the information desired in the 
areas selected for his special investigation, and 
either the chairman or the team as a whole 
should lay specific plans for obtaining required 
data on overall aspects of the program. This 
specific planning should include consideration of 
what areas and data will be observed, which in- 
dividuals (by job assignment) will be contacted, 
and what lines of questioning (and, to some ex- 
tent, which specific questions) will be pursued. 
To this end, the team member may make up his 
own checklist or may modify some standard 
checklist to his purpose. Most of the task area 
treatments in chapter 5 of this text contain 
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pertinent questions. In addition, other docu- 
ments, such as references 3 and 4, include, 
among other things, checklist questions from 
which the surveyor may draw. However, in the 
case of any checklist question, judgment type 
answers to the questions are meaningless unless 
the surveyor is in fact competent to make a 
sound judgment in the particular area in ques- 
tion. Accordingly, where there is doubt, the 
surveyor may elect to orient parts of his own 
survey solely to the collection of facts which can 
be better evaluated and judged later in connec- 
tion with findings of other members of the sur- 
vey team. 

(4) Conduct of Survey Visit . — An intensive 
discussion of conduct of the survey in such as- 
pects as detailed planning, psychology of inter- 
viewing, operational detail (security clearance, 
physical routing within the plant, etc.), entrance 
and exit interviews , and specific formal report- 
ing is beyond the scope of this text. However, 
some basic considerations are worth mention- 
ing. 

It is of paramount importance that the sur- 
vey be conducted in a well- organized fashion to 
obtain the information it seeks in a thorough but 
efficient manner. This not only facilitates an 
effective survey effort but also ensures that the 
survey team causes the minimum practicable 
interference with the contractor’s project effort, 
consistent with survey objectives. To this end, 
the survey itself should be scheduled, and ef- 
fort should be made to adhere to the schedule by 
avoiding ’’interesting,” though not directly ap- 
plicable, digressions— particularly those sug- 
gested by the contractor. However, schedule 
adherence should never be used as an excuse 
for stopping the pursuit of pertinent data if this 
is found to require more time or effort (within 
reason) than originally anticipated. 

(5) Utilization of Findings .— In a number of 
respects, the proper utilization of survey re- 
sults must be a corollary to the basic function 
of the survey and its initial planning. It should 
suffice here to state that the survey has the pri- 
mary purpose of aiding in the evaluation of the 
contractor’s reliability effort (to guide correc- 
tive action where appropriate). It also serves 
a secondary purpose of detecting where the re- 
liability monitoring and evaluation effort re- 
quires reorientation, further study , or strength- 


ening. Accordingly, the results of the survey 
should be reports that document findings and 
recommend appropriate actions. 

Special On-Site Evaluation Studies 

In the overall evaluation program it will be 
necessary to make a detailed study of one or 
more aspects of the contractor’s reliability 
program or of some of the tasks which the re- 
liability program monitors. Examples of such 
areas are the failure-reporting system, the de- 
sign specifications, the failure mode, effect, 
and criticality analyses (FMECA), and quali- 
fication status of hardware. Normally, a 
study of this type is primarily done to assist 
the contractor in putting the area in question in 
good order, rather than being limited to fact- 
finding and evaluation. Frequently, these stud- 
ies will require a temporary residence in the 
contractor’s facility for several weeks or a 
longer overall period involving time at both the 
plant and the NASA center. 

This type of effort concentrates on correct- 
ing weaknesses and on showing the contractor 
how he himself can conduct a better task or pro- 
gram. It should therefore be strongly favored 
by the evaluation group wherever applicable. 
Personnel involved need not be solely from the 
reliability assurance area; where manpower is 
limited, the use of contract effort by an inde- 
pendent study team may be considered to aug- 
ment the special study team. 

Evaluation and Followup 

The final form of evaluation participation by 
the personnel at the NASA center is both techni- 
cal and managerial. It involves not only the 
integration of all the reliability inputs into a 
coherent picture and the judgment of the extent 
to which that picture fits current project needs, 
but also the more difficult judgment and rec- 
ommendation to the NASA project manager of 
the appropriate action to take. Many of the fac- 
tors involved in these functions are discussed at 
various places throughout this text, but these 
can only serve as guides to the overall evalua- 
tion. Assuming reasonable resources and sup- 
port, the ultimate success of the evaluation ef- 
fort will depend primarily on the effectiveness 
of those directing it and their realism, astute- 
ness, and industry in approaching the evalua- 
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tion of the reliability program in terms of the 
overall project situation. 

DELEGATION OF EFFORT 

All reliability program evaluation effort 
must either be performed by the NASA center, 
be contracted, or be delegated. Although dele- 
gation of efforts may possibly be debatable from 
a theoretical viewpoint, practical requirements 
of most contract situations will make it ad- 
vantageous to delegate as much of the evalua- 
tion effort as can be done meaningfully. This 
last qualification— that it be meaningful — is an 
essential one. Delegation beyond this extent is 
one of the chief causes of a poor overall evalu- 
ation effort. 

Potential delegates fall into three broad 
categories: 

(1) On-site field representatives of gov- 
ernment agencies other than NASA 

(2) NASA plant representatives 

(3) Independent study contractors. 

The status of the first of these differs 
rather sharply from that of the last two in that 
the latter are under close control of the NASA 
center, while control of the former is a much 
more formal matter. There is also a general 
difference in the levels and types of personnel, 
but this may be less true in some specific 
cases. The net effect of these factors is that 
they will strongly influence the number and 
types of tasks that can reasonably be given to 
each type of delegate. 

The basic rule for delegation of reliability 
program evaluation tasks is that no group should 
be given an assignment beyond its technical 
capability, manning ability, or willingness to 
execute as delegated. Other considerations are 
duration of prior residence, spectrum and levels 
of technical talent available, nature of prior 
functions as an organization, motivation and as- 
tuteness of individuals , and a willingness not only 
to accept, but to perform delegated functions 
as required for NASA purposes. Because of 
this variability, the only valid guide for relia- 
bility program monitoring delegations is to make 
a thorough prior investigation of the case at 
hand and judge on that basis what can be dele- 


gated beneficially and to whom it should be 
delegated. 

Liaison 

One of the basic considerations indelegating 
tasks under the reliability program evaluation 
effort is that the delegate becomes, functionally 
speaking, a member of the NASA team. Al- 
though his function may not be a very large one 
in the overall team effort, it is an important 
one. To make him aware of this and to obtain 
maximum benefit from his services, it is nec- 
essary to establish a close rapport between him 
and the central NASA evaluation management. It 
is therefore important, when planning the dele- 
gation of any effort, to plan for the manner of 
liaison with the delegate. This also applies to 
individuals from the NASA center who may be 
on extended detail duty in the contractor's plant. 

The importance of this liaison is much 
greater than is generally realized because it is 
the only effective defense (except reassignment) 
against the tendency to "go native," that is, the 
tendency to sympathize unconsciously to a pro- 
gressively increasing (and inappropriate) degree 
with the contractor's problems and point of view 
as the period of residency in the plant extends. 
This is in no way a reflection on the integrity of 
these resident personnel, but an observation on 
the usual human reaction to a physically remote 
situation as compared with the reaction to the 
problems close at hand. A thorough program of 
liaison, with appropriate emphasis on the true 
dependence of the overall evaluation effort on a 
thorough and objective job by the delegates will 
serve both to keep their sense of identification 
with the "team" and to make them plainly aware 
of their importance in the evaluation effort. 

NASA Plant Representatives 

At first thought , one might view the NASA 
representatives who are resident in the hard- 
ware contractor's plant as the preferred group 
to handle all delegated reliability monitoring and 
evaluation tasks. This group is responsive to 
the NASA point of view, knows the project and 
contract effort intimately, and normally in- 
cludes a number of technical personnel of re- 
spectably high levels of professional competence. 

However, in practice it is rarely appropri- 
ate to try to assign all on-site evaluation func- 
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tions to the NASA plant representative's office. 
In fact, except under special circumstances, 
only a limited number of reliability evaluation 
functions should be assigned to this group, and 
these should be carefully selected. The reason 
for this is the nature and perspective of the 
resident group's basic job, which relates pri- 
marily to functions of on-site contract manage- 
ment in those cases where delegation of these 
functions has been considered and found to be 
inappropriate. This normally dictates that they 
perform both business-type functions and cer- 
tain technical tasks, the orientation of both be- 
ing to act for the cognizant NASA center in mat- 
ters involving the day-to-day operations of the 
contract effort where prompt on-site NASA par- 
ticipation is essential to smooth overall pursuit 
of that effort. Therefore, any reliability as- 
surance tasks delegated to this group should 
normally fit the description of the basic mis- 
sion of the resident representative's office. 
Delegated reliability functions outside this scope 
will frequently have to be relegated to a low- 
priority status , with the result that they cannot 
be accomplished in the keen manner required 
for purposes of the reliability program evalu- 
ating effort. 

The foregoing general discussion of the 
resident NASA plant representative's office re- 
flects the usual case for major contracts. For 
lesser contracts, there may well not be a resi- 
dent representative's office, or it may not be 
able to justify more than a very few technical 
men on its staff. However, there are special 
cases in which, for sound management reasons, 
the resident function has been staffed in depth. 
Under such circumstances it is usually within 
the objectives of expansion of the resident rep- 
resentative's function, and very much in the in- 
terest of the NASA project reliability assurance 
function at the center , to include reliability and 
quality assurance specialists among the per- 
sonnel of the NASA resident group. The num- 
ber and extent of reliability program evaluation 
functions which should be delegated to this group 
will vary with circumstances. However, a good 
basic principle to observe in use of these per- 
sonnel is to assign them the more vital of the 
delegatable evaluation functions. These might 
include reviewing some of the "for NASA re- 
view" documentation generated by the contrac- 


tor (see NPC 250-1 ((ref. 1)), par. 1.6) or other 
tasks where higher levels of technical and 
project-oriented judgment are required. It 
should be borne in mind, however, that when- 
ever project "emergencies" arise these person- 
nel must perform as project men first and re- 
liability assurance men second. Therefore, 
some flexibility should be provided by the cog- 
nizant Center both in avoiding assignment of too 
heavy a workload and in providing alternative 
evaluation support during emergency periods. 

On-Site Government Agency Field Representatives 

As used here, the term "on-site govern- 
ment agency field representative" means a gov- 
ernment employee who is employed by an agency 
other than NASA to perform contract adminis- 
tration functions in a resident capacity in the 
contractor's facility. These government field 
representatives are probably the most uni- 
versally available of all the potential reliability 
program monitoring sources. Normally, they 
handle a number of contract administration 
tasks, including inspection functions associated 
with the quality assurance effort. Their usaas 
NASA delegates, in general, is established in 
basic government policy agreements, and there 
are detailed mechanisms and procedures for this 
use (see also NASA PRD 65-9, ref. 5). In the 
area of quality assurance functions , the use of 
these groups is governed by NASA quality pub- 
lication NPC 200- 1A (ref. 6). Many of the re- 
quirements and some (but not all) procedures of 
that publication for quality assurance functions 
should reasonably be considered as also appli- 
cable to the delegation of certain reliability pro- 
gram monitoring tasks to those agencies . Since 
that information is already available to the re- 
liability program evaluator, this text will limit 
its scope to discussing the principal advantages, 
disadvantages, and necessary cautions attend- 
ant to the effective use of these groups. 

Monitoring Functions 

The principal area in which the government 
agency field groups can assist the reliability 
program evaluation effort is in monitoring of 
task outputs which are highly specific in nature. 
The variety of tasks in the reliability program 
will usually call for a sizable number of indi- 
vidual outputs. Although many of these outputs 
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are not required to be delivered to the cus- 
tomer , each is needed for the program and must 
be completed on schedule (relative to other 
project events) to be fully useful. The govern- 
ment agency group is very well situated to 
monitor this area and can report on whether 
items were issued and whether they were is- 
sued on time. They can also report on whether 
review meetings were held, who participated, 
and whether reports were generated within pre- 
scribed time limits. 

Other assignments might include (1) moni- 
toring of testing, (2) monitoring of proper 
maintenance of equipment logs (which may come 
under the quality assurance delegation) , or even 
(3) judging adequacy of some reliability task 
outputs if a standard which clearly describes 
the demarcation between adequacy and inade- 
quacy can be provided. 

There are also other areas in which the ad- 
vice of the delegate on appropriate aspects of 
the contractor’s reliability effort is valuable as 
one of the inputs to the overall evaluation. How- 
ever, much of the technical evaluation of a con- 
tractor’s reliability effort must involve NASA 
project trade-off decisions and is therefore not 
appropriate to delegate. Other items which are 
basic NASA responsibility and which should not 
be delegated include final judgment on accepta- 
bility of program plans, test proposals , or other 
cost-type items. 

Reliability Study Contracts 

One rather versatile means of augmenting 
the evaluation of the contractor’s reliability 
program is the reliability study contract. Al- 
though the study contractor cannot be used as a 
monitor of contract compliance as such, he can 
be used to make studies , in-plant observations , 
and analyses in the role of an advisor and con- 
sultant . 

In order to perform the functions envisioned 
here, the contractor must be a bona fide inde- 
pendent study organization, that is, one which is 
not affiliated with an aerospace hardware con- 
tractor. Although extenuating circumstances 
have permitted the existence of a few exceptions , 
it is an almost mandatory rule that selection be 
limited to independent contractors in order to 
preclude conflict-of-interest problems. 


One basic approach for the reliability con- 
tract study is to direct it toward analysis of the 
design, essentially as a basis for an independent 
reliability assessment. Another approach is to 
use it as an independent search for sources of 
unreliability through analysis either of the de- 
sign itself or of the integrity of the procedures 
and controls used in the design and development 
of the hardware. The study contract must be 
oriented toward predefined objectives and tasks 
which fill a specific need identified in early 
planning of the reliability program evaluation 
effort. Also, although it will normally be per- 
formed within the hardware contractor’s fa- 
cility, it should be conducted with minimum in- 
terference with the hardware effort. 

The study effort may be used in part to as- 
sist the hardware contractor to improve his re- 
liability program or to assist the NASA evalua- 
tor in specific areas. However, it should not 
be used as a roundabout means of obtaining extra 
personnel to fill needs ”as they arise” or as a 
substitute for a properly managed reliability 
program evaluation effort. 

Depending on program needs and the ob- 
jectives of the study, the ’’search- for-unrelia- 
bility” type of contract might study one or 
several of the following areas: 

Reliability and quality assurance programs 

Design 

Fabrication 

Test and checkout 

Handling and shipping 

Launch operations 

Sources of potential unreliability within 
these areas with which the study might be con- 
cerned include: 

Mission profile factors 
Potential failure modes 
Procedures and interfaces 
Other human- error sources 
Implementation of procedures 
Program and configuration controls 
Program communication and feedback 

Usually, when the study is oriented toward 
searching out potential sources of unreliability 
it is highly desirable to utilize a rapid, informal 
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reporting method to facilitate the initiation of 
early corrective action where weaknesses are 
found. 

Some advantages of using reliability con- 
tract studies are listed below: 

(1) Reliability contract studies enable the 
evaluation effort to buy the services of a team of 
specialists in the required areas on short notice 
and for a definite period of need. Thus, where 
specific reliability-critical problem areas re- 
quiringtechnical study in depth can be identified, 
the reliability program evaluation function can 
use the study to obtain proper coverage when 
NASA center specialists cannot be spared for the 
job. 

(2) Once the contract has been executed and 
the contractor's charter is clearly established, 
he is in a position to study, observe, evaluate, 
and report directly and promptly to the contract 
supervisor with a minimum of organizational 
red tape within either the organization of the 
customer or the hardware contractor. 

(3) Usually, the study contractor’s self- 
concern for maintaining his reputation motivates 
him to perform to high standards of objectivity 
and competence. 

The use of study-contract assistance has its 
attendant problems. Some of these are as fol- 
lows: 

(1) Contractors must be selected very 
carefully. Not only must the contractor be well 
qualified as a firm, but the team of specialists 
he provides must be well qualified and well 
managed. Otherwise the advantages of the con- 
tractual study are largely negated. 

(2) The study contract is a cost item that 
must be financed with project funds. It must, 
therefore, be justified in advance and must 
compete with other project needs for its funding. 

(3) Contractual assistance in advisory 
areas must be used cautiously. The NASA 
evaluator must be careful to employ the con- 
tractor’s findings as one of the inputs to the 


overall NASA evaluation rather than simply to 
let them become the complete evaluation. 

The final decision as to the use of a study 
contractor, like so many of the other elements 
of reliability program evaluation, must be 
based on a trade-off of considerations involved 
in the situation in question. However, the study 
contract approach does merit consideration 
wherever it appears that the sum of the other 
available evaluation resources cannot effectively 
cover all necessary areas of reliability program 
evaluation. 

RELIABILITY EVALUATION IN SMALLER PROGRAMS 

The foregoing paragraphs in this section 
have presented the uses of all potential evalua- 
tion resources and described a rather complete 
and sophisticated organization of the evaluation 
effort. The intent has been to show a relatively 
complete catalog from which the evaluator of 
any specific reliability program can select or 
adapt items to fit his program requirements and 
resources. 

It should be recognized that for smaller 
projects and for some of intermediate size the 
total range of evaluation resources described 
here is seldom available. Further, many proj- 
ects of this type do not have available at the 
NASA center reserves of reliability personnel 
and other technical specialists as large as those 
implied by some of the descriptions of NASA 
center evaluation functions. In such situations 
the evaluator must revise the approach to make 
maximum use of evaluation methods that have 
the most benefit on a day-to-day basis, with a 
limited use of the more formal and sophisticated 
approaches which require large levels of effort 
at periodic milestones. This revised approach 
should be extended directly, as much as possi- 
ble, to the evaluation of subcontractors as well 
as of the prime contractor. Otherwise, the lack 
of resources to mount a large-scale formal 
evaluation effort may permit reliability program 
problems, particularly at the subcontractor 
level, to persist too long without detection. 


CHAPTER 5 


Reliability Program Elements and Their Evaluation 


Chapter 3 provided a foundation for gen- 
eral criteria to be used in evaluating reliability 
programs . Chapter 4 gave detailed guidelines 
for NASA participation in, and monitorship of, 
these programs and provided an insight into a 
general technique for evaluating contractor op- 
erations. With such a basis, it is now appro- 
priate to elaborate in some detail on the in- 
gredients of an adequate reliability program 
and to discuss the application of these general 
evaluation principles in assessing the individual 
tasks of contractor reliability programs. 

This chapter describes the character of in- 
dividual elements that should be considered for 
implementation in contractor reliability pro- 
grams. Each reliability program element is 
discussed in terms of its contribution to the 
the project, the responsibilities of the con- 
tractor in providing the element, and how and 
to what level the evaluation must be carried in 
order to make a satisfactory judgment of ef- 
fective contractor implementation. 

In applying this information to particular 
projects, the NASA evaluation personnel must 
be cautioned to treat this material as guideline 
information which shows the thought process 
necessary to develop a sound evaluation tech- 
nique; it is not a substitute for adequate specific 
interpretation of project requirements. 

5.1 RELIABILITY PROGRAM AND ITS ELEMENTS 
Relationship to System and Mission 

The overall objective of the reliability pro- 
gram is to help improve the capability of the 
system hardware (and software) to accomplish 
successfully the space mission for which it is 
developed. Therefore, one of the prime con- 
cerns in each reliability program task area is 
the assurance not only of the adequacy of per- 
formance of each component and subsystem as a 
separate unit but also of the integrity of the 
functioning of all of them together as a com- 


plete system. Further, the system concept can- 
not be limited to the consideration of launch 
vehicle, spacecraft, and associated ground and 
test equipment (of an immediate nature) ; it must 
also extend to the area of mission operations 
wherein the spacecraft (and to a lesser extent, 
the launch vehicle) is tracked, commanded, and 
controlled from earth, and data are received 
and reduced. This latter area involves its own 
equipment, much of which is not limited in its 
use to the support of any one project or mission. 
It also involves heavily the use of operations 
personnel and software (computer programs and 
various procedures). 

Obviously, hardware, communications, or 
human failures in the mission operations system 
can conceivably be as fatal to accomplishment 
of the space mission as failure of the spacecraft 
itself. Therefore, the reliability program must 
operate with an awareness of the mission sys- 
tem as a whole (flight hardware system and mis- 
sion operations system). Although the flight 
hardware contractor may not have responsibility 
for building or operating the mission operations 
system, his project effort, including his relia- 
bility program, must be planned and conducted 
to take into account the capabilities and limita- 
tions of that system. Moreover, the project 
personnel (including reliability personnel) at 
the NASA center must give particular attention 
and support to this vital interface, and the re- 
liability program evaluation effort must give 
appropriate attention to this aspect in the evalu- 
ation of the contractor's reliability program. 

Elements of Reliability Program 

Although a proper evaluation of the con- 
tractor^ reliability program must take an over- 
all system and mission viewpoint, the evalua- 
tion is most feasibly approached through 
consideration of the separate elements and in- 
terfaces of the hardware and the treatment of 
these and their synthesis in separate reliability 
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program tasks or elements. These tasks are 
divided here as follows: 

(1) Reliability program management 

(2) Training and indoctrination 

(3) Subcontractor and supplier control 

(4) Design specifications 

(5) Reliability prediction and estimation 

(6) Failure mode, effect, and criticality 
analyses (FMECA) 

(7) Human engineering and maintainability 

(8) Design review program 

(9) Failure reporting and correction 

(10) Standardization of design practices 

(11) Parts and materials program 

(12) Equipment logs 

(13) Testing and reliability evaluation. 

Documentation of the reliability program is 
not considered here to be a separate element of 
the program as such. Rather, it is recognized 
that each of the elements listed above requires 
documentation for transmitting the results of 
work performed in the element, for technical 
record purposes, and as an internal tool for the 
project. Some factors to be regarded concern- 
ing the proper perspective for viewing and eval- 
uating the function of program documentation 
are given in the concluding section of this 
chapter. 

The attributes of effective implementation 
of each of the above-listed elements will be dis- 
cussed in the succeeding sections, along with 
the principles for the monitoring and evaluation 
of each element by the customer. Incorporation 
of these details into a general program- 
evaluation approach, staffed and managed in ac- 
cordance with the principles discussed in the 
preceding chapter, should result in an effective 
reliability program evaluation. 

5.2 PROGRAM MANAGEMENT 
Management Structure and Responsibilities 

Where complex hardware systems are be- 
ing developed, a loosely organized reliability 
effort is inadequate for obtaining successful re- 
liability assurance. Therefore, it is necessary 
to organize all reliability support functions into 
a coherent program and to provide that program 
with its own control and management. The 


function of reliability program management , as 
with any management function, is basically as 
follows: 

(1) To develop the necessary program 
planning 

(2) To establish, operate, and control the 
organization responsible for implementing the 
program 

(3) To monitor and ensure effectiveness of 
the program. 

Since the purpose of the reliability program 
is to support the project, the reliability pro- 
gram management function will normally re- 
port directly to the overall project management. 
In this way it can best provide timely inputs for 
effective project decision-making. The respon- 
sibilities and authority of reliability program 
management should be clearly specified in 
authoritative project management directives 
and in a reliability program plan (RPP) . 
Ideally, the RPP should be issued as a formal 
project instruction and clearly cited as the gov- 
erning document for reliability program imple- 
mentation. Elaboration of the responsibilities 
of the reliability program management element 
is as follows: 

(1) Planning— the establishment of a work- 
able reliability program plan, including the 
budgeting of manpower and resources, and 
scheduling of program support outputs in ac- 
cordance with the anticipated best utilization by 
the project. 

(2) Operation and Control— the development 
of an organization to implement the program 
plan, and control of the scope and procedures 
of that organization through program directives, 
meetings, and other techniques. 

(3) Program Monitoring— the continuous 
monitoring of reliability program costs and ef- 
fectiveness, and feedback of results of this 
monitoring effort to ensure that the program is 
providing maximum support in proportion to the 
resources available. 

Normally, a management structure will 
have been agreed upon by NASA and the contrac- 
tor during contract (or reliability program) ne- 
gotiation. Revisions to the original reliability 
program management structure may be required 
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as the program progresses and as project situ- 
ations demand changes in program emphasis. 
At all times the management setup should be 
one that will provide best control of the pro- 
gram and maximum input to the project manage- 
ment, rather than one dictated by less practical 
considerations (regardless of how good they may 
seem in theory). 

Regardless of its structure, reliability 
management must take the initiative in provid- 
ing status information to project management ; it 
must keep current with the effectiveness of its 
support program and must transmit information 
to the project with appropriate authority when 
hardware conditions or operations that have an 
adverse effect on mission reliability are de- 
tected. Above all, the reliability program cov- 
ers a wide diversity of tasks, many of which 
are not under the direct control of the reliability 
program manager; it is therefore all the more 
important that the reliability program manage- 
ment be as strong and astute as possible to cope 
with those demanding responsibilities. 

Evaluation of Management 

Evaluation of effectiveness of reliability 
program management in steering the program 
and providing useful inputs to project manage- 
ment will normally be derived from evaluation 
of the individual reliability tasks. Such man- 
agement weaknesses as lack of control or lack 
of sufficient backing by project management for 
important reliability support functions will be 
evident to the evaluator in his monitoring of the 
implementation of other reliability program 
tasks. Seldom can the evaluation be effectively 
accomplished by separating the management 
element from the reliability program and per- 
forming an independent assessment based on 
what theoretically comprises satisfactory man- 
agement practice. 

The evaluator should, however, ensure that 
the basic management framework of planning, 
control, and monitoring activities discussed in 
the previous section is specified in an RPP or 
project directive. He should review this plan 
(if he has not already been closely associated 
with its content) and determine the following: 

(1) Whether the management responsibili- 
ties are clearly defined. Any vague general 


statements that prevent clear understanding 
should be documented and corrected. 

(2) Whether a schedule of reliability pro- 
gram outputs keyed to major project-decision 
points has been established. This is the first 
step in maintaining a program that is oriented 
to project needs rather than one that operates 
independently as a mere academic exercise. 

The contents of the RPP will then guide the 
investigation of each program element. During 
the evaluation of these elements, the evaluator 
should look for the following types of evidence 
to obtain a measure of managements effective- 
ness: 

(1) Has management established all pro- 
gram elements called for in the RPP? If not, 
are the discrepancies between the RPP and es- 
tablished organization explainable by a proven 
lack of need at the time or by a lack of funds or 
personnel? 

(2) Does the lack of any particular element 
constitute a recognizable loss of effectiveness of 
the program? 

(3) Is reliability management making an ef- 
fort to keep the program focused toward areas 
of maximum project need ? 

(4) Is there any indication that management 
has issued timely directives to revise proce- 
dures and responsibilities where a lack of pro- 
gram impact has been noted ? 

(5) Does it appear that reliability manage- 
ment lacks sufficient voice to project manage- 
ment in pointing out reliability problems ? Is 
there a lack of recognition by project manage- 
ment of reliability problems , as compared with 
other trade-off and decision-making informa- 
tion at their disposal? 

(6) Is there evidence that the various re- 
liability subgroups lack a clear understanding of 
their interface with each other and with other 
contractor functions? Reliability management 
should provide a proper definition of responsi- 
bilities and establish communication links 
through meetings, memos, and informal per- 
sonal contact. 

(7) Do the personnel in supervisory posi- 
tions maintain full schedule consciousness in all 
tasks within their responsibility? It may be 
necessary in many instances to lower the level 
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of detail of certain tasks in the interest of pro- 
viding important information to the project when 
hardware decisions must be made. More spe- 
cific details can often be added later. 

(8) Is there a lack of formal or informal 
status reporting within the reliability organiza- 
tion by which reliability management can main- 
tain close contact with all current hardware 
problems ? 

(9) Has reliability management succeeded 
in creating an attitude that the reliability sup- 
port function has value to offer to the project ? 
This attitude must, to a large extent, grow out 
of the daily performance of the reliability 
groups, but it is the overall responsibility of 
management. Techniques for creating this at- 
titude through training and motivation are dis- 
cussed in the following section. 

This type of subjective evaluation obviously 
requires a continuous monitoring and appraisal, 
but it is the best approach for determining the 
real value of management operations from the 
standpoint of quality of performance, timeliness 
of action, and program impact. 

It is difficult to measure management quan- 
titatively in relation to cost effectiveness . While 
other reliability program functions can be elimi- 
nated when they are unproductive, a management 
function must always exist. Its value cannot be 
related to cost (except when it is obvious that an 
unwieldy, top-heavy reliability management 
structure exists). Rather, it must be con- 
tinuously evaluated for its ability to project the 
importance of reliability into project operations 
and to direct the reliability assurance efforts 
toward providing maximum project support. 
When the reliability management function is ef- 
fectively discharging this responsibility, its cost 
is justified. 

5.3 TRAINING AND INDOCTRINATION 
General Considerations 

Reliability training is normally less de- 
pendent on project factors and is more a func- 
tion of management decision by the customer 
than any other element in the program. The 
general NASA requirement for training, stated 
in NPC 250-1 (ref. 1), recognizes in principle 
a government responsibility to bear the cost of 
training in reliability problems peculiar to the 


project. However , in practice , a wide range o“f 
training efforts is encountered, largely because 
contracts are awarded on the basis of many fac- 
tors, of which reliability management qualifica- 
tion is only one. When the selected contractor 
is considered weak in some areas of reliability 
program capability, customer project manage- 
ment will often elect to fund the cost of strength- 
ening these areas through appropriate training 
or to sponsor motivational indoctrination. 

Accordingly, the reliability program evalu- 
ator must guide his judgment of the contractor's 
training effort primarily on the basis of the fol- 
lowing: 

(1) Determining the scope of training re- 
quirements stated in the contract 

(2) Determining whether effort expended 
within that scope is effective and efficient 

(3) Recommending changes in that scope 
based on very clear indication of training needs 
evidenced by overall reliability program per- 
formance (or the need to prune an ineffective 
training effort from the reliability program). 

Training Program Requirements 

The training element of the reliability pro- 
gram is one that falls in the category of a man- 
agement function in that it is separated from 
the mainstream of project planning and control 
activities. It should be directed toward edu- 
cating appropriate employees in special tech- 
niques in the technical disciplines involved in 
performing reliability tasks and educating other 
appropriate employees to recognize the potential 
benefits of the reliability program to the project. 
It should also be directed toward motivating all 
program personnel in the importance of design- 
ing and manufacturing reliability into the hard- 
ware. Because of the varying groups of person- 
nel and disciplines within the contractor's 
facility, this activity must be approached on a 
fairly broad front, with each facet of the train- 
ing program specifically tailored for the group 
involved. It should be emphasized, however, 
that a training program must be both selective 
in the number and type of trainees involved and 
cost effective in the value received from the 
total training expenditure. The approaches to 
training of the principal groups are discussed 
in the following sections. 
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Systems Engineers 

Training of systems engineers (and project 
engineers) should focus primarily on the use of 
the reliability program as a means of effecting 
discipline from a system-wide technical man- 
agement viewpoint. It should emphasize those 
tasks and aspects which contribute to (1) basic 
system planning (e.g. , reliability review of 
mission profile studies and design specifica- 
tions; FMECA), (2) control of interfacing be- 
tween units, (3) system trade-off decisions, and 
(4) analysis and assurance of the design from 
the viewpoints of the system and mission as a 
whole. 

Design Engineers 

Training of design engineers should cover 
briefly the same areas as that for system engi- 
neers. However, it should place primary stress 
on the benefits to be derived from proper inte- 
gration of reliability task outputs into the design- 
development effort, particularly at the hardware 
unit levels. By recognizing the past lack of ac- 
ceptance by design personnel of some support- 
group inputs to the design function, the training 
should aim at showing graphic and convincing 
ways in which the reliability task output can be 
of material assistance to the design effort. 
(Such areas as FMECA and proper parts appli- 
cation are of particular interest here.) It should 
also be quite frank about the limitations of the 
usefulness of these outputs. 

Reliability Engineers 

The training for reliability engineers should 
be aimed at teaching the use of new or special 
techniques required by the project in such areas 
as mathematical modeling, reliability block 
diagramming, redundancy analysis techniques, 
circuit design analysis, or failure mode, effect, 
and criticality analysis. It should not be 
directed toward teaching so-called specialists 
the rudiments of their own area. This training 
may be designed around existing text material, 
if such is available, or around manual material 
especially written for the project at hand. This 
should be augmented with instruction by relia- 
bility engineers experienced in application of 
these new techniques in order to enable the 
trainees to absorb and apply them promptly. 


The selection of participants in this train- 
ing should be limited (in most cases) to em- 
ployees actively engaged in reliability engineer- 
ing and, furthermore, should be directed to 
those of lesser experience who can be expected 
to benefit from it. In some cases, design engi- 
neers working directly with reliability problems 
might also be included, particularly in such 
areas as failure mode, effect, and criticality 
analysis. 

Management and Administrative Personnel 

For the contractor's management personnel 
and for select administrative personnel in perti- 
nent functional areas , the reliability indoctrina- 
tion approach should stress the same areas as 
for design and systems engineers. However, 
the treatment should be more concise and less 
technical. It should also emphasize economic 
aspects and the contributions of the reliability 
program to, and its interfaces with, other 
areas of the contract effort as a whole. 

Motivational Material 

The motivational material must provide an 
effective message to all plant personnel, both 
production workers and those in the various 
professional activities, emphasizing the role of 
each in helping to assure project success. 
Typical techniques include a program of movie 
and slide presentations, periodic lectures, 
periodic leaflet handouts, and posters. To be 
effective the motivational program must be 
dynamic. The material (particularly posters) 
must be frequently changed to provide a con- 
tinuously fresh outlook on the reliability prob- 
lem, in order to focus attention on situations 
applicable to the project while generally retain- 
ing an air that reliability is everyone T s job. 

Evaluation of Reliability Training 

The evaluator should be concerned from 
the outset with determining whether the training 
program is providing a useful service to the 
project. The reliability educational and moti- 
vational program can be a valuable asset if it 
aims a particular message to the groups of 
people most in need of being informed. 

It is important to the efficiency of the re- 
liability training effort that selection of person- 
nel for training be made properly, particularly 



28 


AN INTRODUCTION TO THE EVALUATION OF RELIABILITY PROGRAMS 


that of those to be exposed to technical relia- 
bility engineering material. The evaluator 
should carefully examine the contractor’s 
trainee selection technique and ascertain that 
the candidates for technical training are of the 
following types: 

(1) Personnel performing or utilizing re- 
sults of a reliability engineering task 

(2) Personnel of lesser experience in the 
application of the reliability discipline who will 
derive maximum benefit from exposure to the 
training program. 

These qualifications are cited to emphasize 
to the evaluator the inappropriateness of train- 
ing programs in which trainess are already 
uniquely qualified in the conduct or use of re- 
liability support work or where they are selected 
for technical reliability training even though 
their project areas do not interface with the re- 
liability support effort. 

In summary, the basic elements of the as- 
sessment of the reliability training effort are 
determinations of the following: 

(1) Are the contractor employees receptive 
to the training material or do they seem to take 
it very lightly? What are their attitudes ? 

(2) Does some of the course material ap- 
pear to be more valuable than some other ? At- 
tempt to rank the value. Are some important 
subjects being overlooked? Are others over- 
emphasized? 

(3) Do the employees appear to have bene- 
fitted from the training; that is, is there any ap- 
parent difference in attitude toward usefulness 
of the reliability tasks or in job proficiency be- 
tween ’’trained” and ’’untrained” personnel of an 
otherwise comparable level ? 

(4) Has the contractor selected personnel 
for formal reliability training based on their 
need for further education? 

(5) Is the contractor providing the best 
program at minimum cost, or are some need- 
less efforts being expended ? 

After making these judgements, the evalu- 
ator should determine if modifications can be 
made, considering the time remaining and the 
contribution of such modifications to program 
improvement. Budget, of course, is always to 


be considered; it may prevent changes that would 
provide improvement. However, decisions 
based on budget alone should ultimately be made 
at a higher management level. 

5.4 SUBCONTRACTOR AND SUPPLIER CONTROL 
Contractor Requirements 

Because of the large number of subcon- 
tractor-supplied components and subsystems 
that accompany nearly all space programs , the 
prime contractor’s reliability program must 
provide controls for assuring adequate relia- 
bility of this ’’bought” hardware. Reliability 
assurance in this area is exercised through the 
following: 

(1) Selection of subcontractors from the 
standpoint of demonstrated capability to pro- 
duce a reliable product 

(2) Development of adequate design speci- 
fications and test requirements on the subcon- 
tractor-produced product 

(3) Development of proper reliability and 
quality program requirements to impose on each 
subcontractor 

(4) Establishment and maintainence of 
close day-to-day technical liaison with the sub- 
contractor (both in design and reliability areas) 
in order to minimize communication problems 
and to facilitate earliest identification and cor- 
rection of design problems of an interface or 
interrelation nature 

(5) Continuous review and assessment to 
assure that each subcontractor is implementing 
his reliability and quality assurance programs 
effectively. 

Reliability assurance requirements must be 
imposed on subcontractors and suppliers on the 
basis of criticality of the hardware item being 
supplied. Similarly, the depth of these require- 
ments should govern the amount of effort ex- 
pended by the contractor to assure that the sub- 
contractor is performing his reliability 
assurance function adequately. 

Where suppliers of major components and 
subsystems are concerned, the prime contractor 
must evaluate each subcontracted item inde- 
pendently to determine the extent and type of 
reliability program needed. He should then 
impose appropriate reliability program re- 
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quirements on each subcontractor so as to pro- 
vide reliability assurance in these areas of 
need. 

Each major subcontractor is required to 
submit a reliability program plan, and the con- 
tractor must monitor the subcontractor's im- 
plementation to assure compliance with the plan 
and to assess the timeliness and adequacy of in- 
dividual tasks as the project advances. The 
subcontract must contain surveillance entrees 
to permit this. This procedure places the con- 
tractor reliability group in a situation very 
similar to that of the NASA center with regard 
to monitoring and evaluating reliability pro- 
gram performance. Therefore, the methods 
and procedures described in other subsections 
of this section, with regard to the manner of 
performing the monitoring and evaluation of 
various program elements, apply to the con- 
tractor-subcontractor relationships in the same 
way that they apply to NASA evaluation of the 
contractor’s program. 

The assurance of reliability of parts and 
components not formally classified as "major" 
must rely on the controls imposed through the 
contractor’s quality assurance program. De- 
creased emphasis in this area is dictated by 
obvious economic considerations. 

Evaluation Techniques 

In assessing the contractor-subcontractor 
relationship, the NASA evaluators must proceed 
in view of the similarity of this relationship to 
the NASA-contractor interface. Their evalua- 
tion of the contractor’s effort in this area should 
be concerned with four basic items: 


(1) The appropriateness of the reliability 
and quality requirements imposed on each sub- 
contractor in light of the significance of the 
hardware item being procured 

(2) The degree of (informal) technical com- 
munication existing between contractor and sub- 
contractor counterpart personnel 

(3) The contractor’s program to monitor 
and evaluate subcontractor reliability efforts, 
with particular emphasis on the apportionment 
of the contractor’s effort within this area 

(4) The contractor’s procedure for moni- 
toring and evaluation of the subcontractor 


Although item (1) will usually have been 
agreed to in basic form in the NASA approval of 
the contractor’s Reliability Program Plan, it 
should not be dismissed in the evaluation, since 
misjudgments sometimes occur in initial plan- 
ning and other factors may also dictatechanges. 

Item (2) is of particularly great importance , 
since a constructive effort here of adequate 
scope can do much to preclude, or at least 
minimize, the seriousness of deficiencies in 
items (3) or (4). 

With regard to item (3) , the evaluation can 
adapt and use the basic guides presented in 
chapter 4 as they would apply to a contractor- 
subcontractor relationship. 

Finally, the NASA evaluator should ascer- 
tain that the contractor is monitoring and evalu- 
ating each subcontractor reliability program 
element in a manner similar to that specified in 
the various portions of this section. Where 
there is doubt of the effectiveness of the prime 
contractor in guiding and monitoring the relia- 
bility efforts of his subcontractors, particularly 
of major subcontractors, the NASA evaluation 
program should give consideration to a first- 
hand evaluation of the subcontractors in question 
(assuming that adequate entrees exist). 

It is important to recognize that the evalu- 
ation of reliability assurance effort on subcon- 
tracted items must be made in the perspective 
of a net project- oriented trade-off rather than 
as a distinctly separated "reliability" activity. 
The reliability program evaluator must, there- 
fore, consider his immediate % valuation to be of 
greatest significance as one of the inputs to the 
NASA project management evaluation of the 
overall subcontractor area. Its contribution to 
the evaluation of the contractor reliability effort, 
by itself, is secondary. 

5.5 DESIGN SPECIFICATIONS 
Contractor Requirements 

For the ultimate achievement of a reliable 
system it is imperative to assure the proper 
specification of all design parameters, such as 
environment, performance, and operation, as 
well as reliability requirements, at the earliest 
possible date. These must be specified first as 
they apply to gross mission functions at the 
spacecraft, launch vehicle , and mission opera- 
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tions levels; these are usually given in project 
definition documents. They must then be speci- 
fied within each gross mission function (e.g. , 
spacecraft) to spell out the functions of its sub- 
elements. It is at these levels (subsystem and 
black box) that design specifications as such 
must be generated to provide detailed design 
criteria for guidance to (1) contractor design 
groups and (2) major subcontractors. These 
specifications are the primary documentary 
link between the establishment of a mission and 
the design of system hardware. A design speci- 
fication is the detailed interpretation of the mis- 
sion in terms of characteristics that define the 
requirements the hardware must meet. Im- 
proper or poorly defined specification require- 
ments can lead to a system that is incapable of 
performing the intended mission. Normally, 
poor specifications will result in redesign or 
"patch-up" of subsystems, a less -than- optimum 
design configuration, probable over -complexity, 
and subsystem interfaces that are inconsistent. 
In general, they will result in low reliability, 
not through limitations of design capability, but 
through failure to define the design problem. 

Because of the vital role design specifica- 
tions play in providing the basis for the design 
of reliable hardware, it is important that they 
be reviewed systematically by a group that is 
independent of those responsible for generating 
them. Ideally, the reliability group should have 
the system engineering and design engineering 
talent necessary toperform this function. How- 
ever, in the many situations where it does not, 
the specification review task may rightfully be 
delegated to some other independent group pos- 
sessing the necessary talent. Regardless of 
who is assigned this function, it is the responsi- 
bility of the reliability group to ascertain that 
an adequate specification review is performed. 
When the technical aspect of the review task is 
delegated, the reliability group should at least 
retain the role of monitoring and auditing to as- 
sure that an adequate set of specifications is be- 
ing generated and updated as trade-offs are 
made at various project decision points. 

Evaluation Considerations 

The preceding discussion indicates that the 
NASA evaluation should determine that the re- 
liability group is: 


(1) Performing an adequate specification 
review and critique, or 

(2) Monitoring the effort of a group that has 
been delegated the specification review respon- 
sibility. 

Either method is acceptable if it results in 
adequate determination that each design speci- 
fication is complete and is kept current and that 
the set of these specifications covers the entire 
system and its interfaces without omission. 

The evaluator should be wary of a meaning- 
less response to the requirement for a review 
function. One of the primary signs of such a 
response is the project situation in which the 
magnitude of the technical review task far ex- 
ceeds the time and level of effort available. 
This type of situation usually forces a meaning- 
less concurrence with the contents of the docu- 
ments in the interest of meeting the project 
schedule. Use of unqualified personnel will 
provide a similar result. Additional evidence 
of this situation will be seen in a lack of any 
constructive critiques and recommendations 
from the reviewing group. 

A distinction should be made, however, be- 
tween the deliberately poor or indifferent re- 
sponse and difficulty encountered in a truly 
conscientious effort to keep up with a demanding 
task under heavy time schedule pressure. Nor- 
mally, the dynamic project situation will not 
afford a "comfortable" amount of time for the 
review. It will therefore be imperative that 
adequate coordination be maintained between the 
specification initiator and reviewing group so 
that early drafts or portions of the specification 
will be made available for beginning the review 
early in the specification draft period. 

The original subsystem design specifica- 
tions can be expected to require revision, par- 
ticularly during the early development phase as 
initial design decisions and trade-offs are made. 
Difficulties may arise in providing the originally 
established outputs of one subsystem within re- 
quired designtolerances; this would necessitate 
revisions to the specifications for the interact- 
ing subsystems. Similarly, weight and volume 
trade-offs between subsystems are common and 
should result in specification revision. The 
evaluator should look for evidence of specifica- 
tion updating during the development period, 
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particularly at major design decision points. 

It must be remembered that the responsi- 
bility for final determination of technical ade- 
quacy of the contractor's design specifications 
is normally assigned to the systems engineers 
or design personnel in the NASA project man- 
ager’s organization. This responsibility would 
include responsibility for judgment of the levels 
and tolerances specified for such parameters as 
signal levels, gains, mechanical stresses, and 
power requirements . The reliability program 
evaluator must rely on the systems engineers 
for assistance in judging suspected weakness in 
the detailed content of design specifications. 

Additional points to pursue in the evaluation 
of the contractor specification review activity 
are: 

(1) Is the contractor using qualified per- 
sonnel to review and evaluate design specifica- 
tions ? This question can often be answered 
through personal contact between the NASA 
evaluator and the personnel in question. 

(2) Does the specification review group fol- 
low an organized approach to its tasks (e.g. , 
schedules and informal status reporting) ? 

(3) Is a design specification of sufficient 
detail drafted for every subsystem and major 
component (black box) ? Does the specification 
properly state the inputs available to the sub- 
system, the output requirements, the environ- 
mental conditions under which it must operate 
(including overstress), other system interfaces, 
reliability goals, and maintainability goals (if 
applicable) ? 

(4) Do the review personnel take the initia- 
tive to consult with the design specification 
group in the early draft stage ? Do the review 
personnel have an orderly method of consulta- 
tion? 

(5) Is there recognition by other project 
personnel that the specification review is 
worthwhile ? 

(6) Is there an effective means by which 
the project takes action as a result of noncon- 
currence by the reviewing group? 

(7) Is the review function responding ade- 
quately within the time constraints imposed by 
project schedule, assuring that the specifica- 
tions contain current information in accordance 
with project developments and are updated as 


required by system trade-offs or mission re- 
visions ? 

(8) Does the complete set of subsystem and 
component specifications describe adequately all 
the mission functions in the system in terms of 
environmental and performance parameters ? 
(Or do significant omissions exist?) 

5.6 RELIABILITY PREDICTION AND ESTIMATION 

Contractor Program 

The analytical reliability prediction from 
which numerical estimates of hardware relia- 
bility are made is one of the basic tools of the 
reliability program. However, this task can 
degenerate into a totally ineffective expenditure 
of effort— or even a serious liability to the re- 
liability program— if the prediction results are 
not supplied to the project in a timely manner, 
if they are poorly done, or if they are im- 
properly used. 

Basically, reliability prediction provides 
quantitative estimates of the reliability of the 
system and its individual subelements. These 
’’numbers" are largely based on a process of 
mathematical synthesis of failure probabilities 
from the subelements of that assembly, usually 
starting from the part level and continuing up to 
the system level . This buildup (model synthesis) 
takes many steps, each of which involves as- 
sumptions. Assumptions are also heavily uti- 
lized in adapting generic failure-rate data to a 
specific project and mission (to apply in the 
model). As a result, the numerical prediction 
of system reliability provides only a broad esti- 
mate of the reliability of system elements. 

On a comparative basis, however, the nu- 
merical indices can usually be quite useful in 
determining the reliability of one system ele- 
ment relative to that of another or in determin- 
ing the reliability status of the system at one 
time in the project cycle relative to status at a 
different time. Because predictions can identify 
weak areas of the design and, furthermore, can 
quantify the weakness of one system element 
relative to that of another, they can serve a 
number of functions, some of which are: 

(1) Helping to guide the selection of one 
design approach over another (at the conceptual 
phase) 
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(2) Guiding apportionment of corrective 
design effort 

(3) Serving as one important criterion for 
the decision on the extent to which it is neces- 
sary to employ redundancy in the design 

(4) Determining the need for additional test 
information, changes in test procedure, and 
changes to test intervals. 

As input data to the prediction become more 
refined and more confidence can be placed in the 
actual quantitative prediction results , the pre- 
diction can acquire usefulness as a decision- 
making tool for higher-level decisions such as 
whether to commit the system to launch. 

One of the most important keys to effective- 
ness of the prediction task is timeliness. Since 
the design-development process for the hard- 
ware is basically iterative in nature (see the 
section entitled "Timeliness" in chapter 3), the 
reliability prediction process must also be 
iterative to support each of the decision points 
in the design-development process. Further- 
more, the prediction must be updated periodi- 
cally, not only to reflect design trade-offs made 
at each decision point but also to replace generic 
and estimated failure-rate data with specifically 
applicable data as they are evolved from the 
project testing program. 

The reliability prediction for all parts of 
the system, including those procured under sub- 
contract, should use compatible prediction tech- 
niques; that is, the same basic ground rules 
must be used for selection of such data as part 
and component failure rates , environmental and 
stress factors, and mathematical modeling ap- 
proaches. The use of compatible ground rules 
is necessary for the prediction results for 
each system element to be compared with those 
for the other elements on a common basis. 

The contractor’s reliability program plan 
should call for early initiation of the reliability 
prediction and estimation element, with an 
initial schedule for periodic updating based on 
the initial schedule of decision points in the de- 
velopment process; and these prediction sched- 
ules should be revised as changes in project 
schedules occur. Furthermore, prediction ef- 
forts should be performed in synchronization 
with design progress. Not only should the pre- 
diction results be provided formally prior to the 


* 

design reviews, but the reliability personnel 
should work closely with design personnel in- 
formally on a person-to-person basis when 
making them. Recommendations for system 
modification to improve reliability can thereby 
be available for consideration at the time of 
trade-off decisions, instead of being too late to 
be of significant value. 

Evaluation Considerations 

Since the key factor in measuring the ef- 
fectiveness of the prediction effort is timeli- 
ness, the evaluator should first look for a 
formal prediction schedule in the reliability 
program plan that is keyed to the overall proj- 
ect development schedule in such a way that 
prediction results will be available in time for 
consideration at major design-decision mile- 
stones. He should be careful to recognize the 
situation in which prediction tasks are being 
performed according to a schedule in the origi- 
nal RPP that has subsequently become obsolete 
because of changes in overall project schedules 
or because of unforseen problems. 

Not only must the evaluator be concerned 
that the contractor is conforming to a schedule 
of prediction efforts and that this schedule is 
kept current but he must also determine whether 
the comprehensiveness of each separate pre- 
diction is appropriate to its intended use. Since 
early predictions are tools for assessing gross 
weaknesses in the design, it is not necessary at 
the early prediction stage to make elaborate 
analyses (such as performing detailed calcula- 
tion of those failure modes considered to have 
minor effect on mission success or waiting for 
data to provide unnecessarily accurate failure 
rates and environmental and stress factors). 
Instead, a gross prediction will often suffice as 
a preliminary design decision input and may be 
necessary to enable it to be used (because of 
schedule pressure). It will at least result in 
early determination of the most unreliable sub- 
systems and suggest those areas which bear 
further immediate investigation. The evaluator 
should look for this type of early program pre- 
diction effort. An attempt to provide the com- 
pletely comprehensive prediction at the outset 
is often the reason for untimely outputs and is 
a fiatural result of poor coordination between 
design and reliability groups. 
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Where the necessary reliability /design co- 
ordination does exist, there should be indica- 
tions that reliability engineers are accomplish- 
ing the following: 

(1) Obtaining engineering drawings before 
their official release 

(2) Sitting in, as necessary to their func- 
tion, on early design meetings 

(3) Providing early advice on areas where 
reliability can be improved 

(4) Obtaining a feel for the depth of analysis 
required at the next decision point. 

This kind of evaluation can best be ac- 
complished by interviewing reliability and de- 
sign engineers separately with questions such 
as those given below which depend on existing 
project status. 

Questions for the reliability engineer are: 

(1) Are you making (or have you made) a 
reliability prediction on the "blank" subsystem? 
Can I obtain a copy of the prediction analysis 
for the most recent milestone ? 

(2) Was the original design concept within 
the preliminary reliability allocation? If not, 
what recommendations were made for improve- 
ment? 

(3) Was your prediction considered in the 
preliminary design review, and were design 
changes made as a result of it? With the under- 
standing that trade-off decisions may result in 
no reliability improvement changes being made, 
what were the reasons for your recommenda- 
tions not being incorporated, if such was the 
case? 

(4) Are you continuing to review design 
changes for their effect on the latest prediction ? 

(5) Is there coordination between the re- 
liability and design personnel in advance of all 
design changes ? Who is responsible for seeing 
that such contact is initiated ? 

(6) Do you review the final drawing change 
release to determine whether it contains the 
same change as discussed in advance ? 

(7) Did you deliver the required prediction 
results at the previous milestone ? 

(8) If the prediction is still in progress, 
will the results be available in time to be con- 
sidered by project management in the next 


design-decision milestone (such as the next de- 
sign review) ? 

Questions for the design engineer are: 

(1) Has your subsystem design been anal- 
yzed by the reliability group and a prediction 
made ? If not, is a prediction in progress ? 

(2) Does the prediction work include all the 
latest design changes ? 

(3) Were you given any recommendations 
by the reliability group as a result of earlier 
prediction efforts, such as the use of redundancy 
and changes in design technique and components? 

(4) Has the reliability group provided rec- 
ommendations that are both practical and in 
time to be of benefit in design decision? Are 
you continuing to coordinate with reliability 
engineering as changes are anticipated ? 

Note that the above questions are all aimed 
at determining the impact on the project of the 
reliability prediction element through its timely 
application of supporting analyses. The evalua- 
tor should be aware that existence of a planned 
prediction requirement does not in itself justify 
unusually large manpower expenditures. The 
levels of effort for prediction, as for any other 
task, must be commensurate with a justifiable 
need. 

Where the evaluator finds that little effort 
or progress has been made in the prediction of 
system reliability, he must determine whether 
to recommend elimination or curtailment of the 
activity, or, if the project is still young enough, 
to suggest increased emphasis or selectivity in 
the prediction effort. The originally anticipated 
needs of the project may no longer apply to the 
existing project conditions . Where a change is 
to be recommended, time is the critical factor. 
Since reliability predictions of certain critical 
subsystems may still be beneficial rather late 
in the program— particularly in support of test 
program decisions such as frequency and level 
of tests— selected prediction effort may be the 
best alternative. 

Nothing specific has been said here con- 
cerning the evaluation of the quality of the pre- 
diction work in terms of appropriateness of 
analytical techniques selected or adequacy of 
their implementation. These factors involve 
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areas such as construction of proper block 
diagrams and mathematical models , application 
of justifiable environmental and stress factors, 
and failure rate determinations. This evalua- 
tion requires astute engineering judgment that 
is best obtained from evaluation personnel whose 
specialty is analytical prediction and who are 
familiar with the latest applicable part failure 
sources and are backed up with adequate com- 
petence in the component and system engineer- 
ing design areas. Their assistance should be 
used to determine that there is a consistency of 
overall methodology by the prime contractor 
and all subcontractors, that failure rate sources 
are compatible, and that selection and employ- 
ment of the specific methodology for each pre- 
diction reflects good judgment and is technically 
sound. 

5.7 FAILURE MODE, EFFECT, AND 
CRITICALITY ANALYSIS 

Contractor Requirements 

Companion to the reliability prediction ele- 
ment is the requirement to perform failure 
mode, effect, and criticality analyses (FMECA) 
of the system hardware. This task is another 
that must be iterative in order to provide design 
guidance during the evaluation of the hardware 
configuration. Together with the reliability 
prediction, the FMECA should provide the pri- 
mary analytical reliability information for de- 
sign and system trade-off decisions at appro- 
priate program milestones. Although it is the 
objective of FMECA to identify all modes of 
failure within the system, its first purpose is 
the early identification of all first-order or 
catastrophic failure possibilities so that they 
can be eliminated or minimized through cor- 
rective design action at the earliest possible 
time. 

The contractor should provide in the relia- 
bility program plan for an FMECA effort that is 
time-phased to correspond with the decision 
points in the development of each system, sub- 
system, and component. Individual analyses 
should be initiated as soon as preliminary de- 
sign information is available on the hardware 
element in question. In order to accomplish 
this overall purpose, the contractor^ relia- 
bility engineers should work with the design 


personnel on a continuing basis in completing 
the following tasks: 

(1) Identifying system, subsystem, and 
component failure modes 

(2) Identifying the effects of potential fail- 
ures on the mission of the system (and relating 
the effects to specific portions of the system, 
such as launch vehicle, spacecraft, crew, or 
ground support) 

(3) Determining probability of occurrence 
of each potential mode of failure 

(4) Recommending appropriate corrective 
features such as redundancy, fail-safe design 
features, backups, and selection of more re- 
liable parts 

(5) Assisting in the formulation of test 
criteria selected in light of identified critical 
failure modes. 

Frequently, it may be necessary and quite 
proper for the design engineer to assume the 
primary role in this analysis at the component 
level because of his complete familiarity with 
the hardware configuration, its operation, and 
the design technology area of the component in 
question. The role of the reliability engineer in 
this case should be one of objective and con- 
structive co-participation. He should provide 
support in use of reliability analysis technique 
and offer an independent point of view on the 
criticality of failure modes and the most effec- 
tive means of lowering their probability of oc- 
currence (including elimination of them) through 
suggested changes in design or mode of opera- 
tion. When making analyses of the subsystem, 
system, and mission levels, the reliability 
group should participate in the FMECA with 
the cognizant system engineering group or, if 
necessary, assume the responsibility for ac- 
complishing the task itself (coordinating with 
the systems engineers as necessary) . 

The results of the FMECA should provide a 
guideline for design reviews to highlight all 
critical first-order failure modes and corre- 
sponding proposed design corrections for scru- 
tiny by the review team. For potential failure 
modes outside the first-order category, the 
probability-of-occurrence data should be use- 
ful to project management in making trade-off 
decisions on the extent of effort and time to be 
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invested in corrective action. Finally, the 
FMECA results should provide criteria for use 
by the contractor’s test personnel in planning 
and designing tests for the system, particularly 
in such aspects as testing depth, test intervals, 
and stresses. 

Evaluation Considerations 

Probably the greatest single criticism of 
past FMECA’s has been the limited application 
of their results for the improvement of system 
hardware. The chief causes of this limitation 
relate to the general considerations of quality, 
timeliness, and cost effectiveness discussed in 
chapter 3 . Some of the specific ways in which 
these appear are: 

(1) Failure to recognize that the FMECA 
must be iterative in nature, to correspond with 
the iterations in the design process itself 

(2) Failure of the RPPto schedule FMECA 
outputs to correspond with scheduled design de- 
cision milestones in the overall project program 
plan 

(3) Failure to keep the FMECA schedules 
current 

(4) Late FMECA reporting 

(5) Lack of confidence of design engineers 
in the value of the analyses and consequent un- 
willingness to use them 

(6) Overemphasis by reliability personnel 
on comprehensive analyses at the expense of 
timeliness 

(7) Use of analysis methods either too 
simple or too sophisticated for the requirement 

(8) Inadequate implementation of the anal- 
ysis method selected, or an isolated effort by 
analytical specialists without adequate technical 
input by knowledgeable design personnel. 

The evaluation should start with the FMECA 
portion of the RPP. Here the evaluator should 
look for a functional schedule (oriented to 
project milestones) of FMECA ’s. This should 
provide for an analysis of each element of hard- 
ware down to the component ("black box") level 
at each appropriate stage in its development. 
For each of these analyses, it should provide a 
clear-cut description of the depth of analysis, 
its purpose, and the planned use of its results. 
The word "appropriate," as applied to the num- 
ber of analyses and their depth, will depend on 


the system element being analyzed, its criti- 
cality in the system, and the nature and re- 
quirements of the system as a whole. Where 
there is doubt in the evaluator’s mind in judging 
this "appropriateness," it is usually best to lean 
in the direction of more, rather than less, effort 
because of the potentially high reliability- 
achievement contribution of the FMECA task. 

In monitoring the implementation of the 
FMECA task, the evaluator should first ascer- 
tain whether the functional schedule of FMECA’s 
is being maintained. This may be done by com- 
paring calendar schedules (past and future) of 
the FMECA’s with the calendar schedules of de- 
velopment milestone events for the correspond- 
ing hardware elements. Not only should the 
original dates and the revisions on the two 
schedules correspond, but evidence should be 
obtained (from project documents, such as 
memoranda) that the changes in the FMECA 
schedules track closely in date of revision with 
those in the hardware schedules. One very 
meaningful way to determine overall timeliness 
of FMECA output is to determine whether it was 
included in the input data package (and in what 
condition) for the corresponding design review. 

The second important aspect of the evalua- 
tion of the FMECA task is to determine the man- 
ner and effectiveness of communicating the 
problem areas discovered in FMECA for early 
design attention. In practice, this determina- 
tion will be closely allied to the schedule de- 
termination (and that of effectiveness) . In the 
ideal case, the reliability engineer will perform 
the FMECA jointly with the component design 
engineer (or systems engineer for FMECA at 
higher assembly levels). In this situation, the 
communication of findings, the attitude of de- 
sign personnel toward these findings, and the 
question of conscientious corrective response to 
them will offer little, if any, problem. How- 
ever, in many cases the reliability engineer, 
himself, will have to conduct the actual FMECA, 
coordinating closely with the design engineer (or 
systems engineer) to obtain design and function 
information needed in the analysis. In these 
cases, the reliability engineer should take the 
initiative, both to obtain this assistance and to 
communicate his findings as expeditiously as 
possible to his design counterpart (but not in 
lieu of formal reporting). The character and 
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effectiveness of this working relationship can be 
best evaluated through interviews of the princi- 
pals , by review of memoranda between the two 
groups , and by determining (through any other 
means available) the points in time at which 
initial corrective design consideration is given 
to FMECA results. Frequently, a detailed dis- 
cussion between the evaluator and cognizant 
NASA project personnel can provide much use- 
ful information on this . 

A sound judgement of the technical quality 
of FMECA outputs is difficult at best because 
each must be judged in light of the nature and 
criticality of the hardware element in question 
and the design milestone which the analysis at 
hand is intended to support. Therefore, the 
evaluator may frequently find it useful or nec- 
essary to obtain additional judgment from quali- 
fied personnel at the NASA center. These per- 
sonnel include design and systems engineers for 
the project and FMECA specialists, who will 
collectively have the background experience in 
the pertinent design technology area, the appli- 
cable analytical techniques, and the intimate 
specific knowledge of the hardware necessary 
for a complete judgment. These personnel 
should review at least a representative sample 
(and hopefully a large porportion) of the FMECA 
reports as they are generated and should pro- 
vide comments to the evaluator on the contrac- 
tor's performance. 

The overall effectiveness of the FMECA 
task can best be assessed through observation 
and analysis of the test program (see section 
entitled "Contractor Requirements’') and the de- 
sign review program. However, this should be 
used as a complement to, but not a substitute 
for, other aspects of the evaluation of FMECA. 
The uses of design review as a vantage point for 
assessing timeliness of FMECA and for follow- 
ing up on corrective design action have been 
mentioned earlier in this section. The func- 
tional roles of the FMECA task and the design 
review task are closely allied. The FMECA, 
on the one hand, identifies first-order failure 
modes for vital corrective design action and 
gives probability-of-oceurrence data to guide 
trade-off decisions on the correction of less 
critical potential failure modes. On the other 
hand, the design review program reviews the 
adequacy of these decisions and actions and the 


logic on which they are based. For this reason, 
the formal FMECA report at each milestone is 
particularly important as a design document. 
Also, when examined together with the design 
review report, it is a prime indication of ef- 
fectiveness of the FMECA. The evaluator, 
should, therefore, devote appropriate attention 
to the formal as well as to the informal outputs 
of the FMECA task. 

In summary, the FMECA task is potentially 
one of the most beneficial project contributions 
of the reliability program. To be effective it 
must (1) competently detect, analyze, and evalu- 
ate all significant failure modes at each perti- 
nent hardware level and milestone, (2) report its 
findings in time and form to be of use at cor- 
responding project decision points, and (3) be 
recognized for its value and be employed ef- 
fectively in improving the mission system. The 
evaluator’s concern will be to ascertain through 
evidence (or absence of evidence) the degree to 
which these functions are being performed. 

5.8 HUMAN FACTORS AND MAINTAINABILITY 
Contractor Requirements 

One of the most significant areas for po- 
tential mission failure or performance degrada- 
tion is the interface between people and machines. 
In. order to provide for reliability assurance in 
this area, the contractor is required to give 
careful consider at ion to the elimination of human 
induced error, particularly in the areas of 
maintainability and serviceability, throughout 
his activities of design, development, and op- 
erational use of the hardware. 

The extent of effort and sophistication of 
approach used in this area will vary with the 
significance and magnitude of the project and 
the nature of the hardware item (e.g. , whether 
or not it is man-operated). However, the basic 
approach to assurance in this area is essentially 
the same in all cases: 

(1) Break down the mission to be per- 
formed into functions . 

(2) Select concepts of hardware to per- 
form each function. In this process, decide 
which functions will be automated and which 
will be manned. Also, at this time establish 
basic maintenance and servicing concepts. 
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(3) As the design and development prog- 
resses through each stage, analyze further to 
bring the maintainability, serviceability, and 
other human factors requirements into sharper 
focus. 

(4) As these requirements are identified, 
respond to each throughout the development ef- 
fort by providing accessibility for maintenance 
and simplification of activities (operation or 
checkout) so that they can be performed with a 
minimum of stress or confusion to the operator, 
by designing equipment to reduce the opportuni- 
ties to misuse or damage it, by devising pro- 
cedures which are clear and effective in di- 
recting activities without error, and by using 
other measures of a similar nature. 

The functions described above are likely to 
be encountered as distinct, separately identi- 
fiable efforts on large programs or on those 
where major system functions are performed 
by human operators. However, they must also 
exist, though possibly on a much simpler scale, 
in intermediate and small-size programs; here, 
instead of a separate formal effort manned by 
specialists, they may be performed variously 
by design, systems, test, and reliability engi- 
neers with little or no assistance from human 
factors or maintainability personnel. These 
functions are described in detail in a as yet un- 
published paper entitled "Introduction to the As- 
surance of Human Performance in Space Sys- 
tems." 

Whatever the extent of human factors in- 
volvement in the design and development of 
flight hardware , the area of assuring human 
performance is vital in the mission operations 
system. This is particularly true during launch 
phase activities in general and for the entire 
mission sequence of lunar or interplanetary 
spacecraft, since there is a high probability of 
losing the mission if human errors in these 
situations are not identified and corrected im- 
mediately. Therefore, development and de- 
bugging of procedures and training to assure 
effective interfacing of these procedures with 
mission operations personnel and equipment are 
a major part of the mission operations system 
activity. 

In summary, whether conducted full-time 
or part-time by specialists or conducted in the 


absence of specialists, the human factors effort 
must identify man- machine interfaces wherever 
they exist and provide appropriate design, in- 
structions, and training measures to minimize 
chances for error. These efforts must be fol- 
lowed through in appropriate assurance activi- 
ties including the design review program, and 
they should be monitored as necessary by the 
reliability management function. 

Evaluation Considerations 

The contractor reliability activity’s role in 
assuring that maintainability and human factors 
are properly considered in the contract effort is 
basically a "checking" function. Therefore, the 
NASA evaluation must attempt to determine 
whether the reliability group is keeping up to 
date on the progress of human factors activi- 
ties and whether those responsible for technical 
support in this area are providing it effectively. 
A full support effort is usually justifiable for 
manned spacecraft, man-rated launch vehicles, 
launch-critical ground support equipment, and 
mission operations equipment and functions 
(particularly for projects of high significance). 
Less emphasis can be expected on projects 
whose hardware interface with the human ele- 
ment is more limited and where mission criti- 
cality is not high. 

The evaluator should ensure that someone 
in the reliability group has the responsibility 
for monitoring and assuring the adequacy of the 
human engineering and maintainability support 
efforts and that he has, in fact, established a 
means for accomplishing this task. Then, to 
determine the effectiveness of the human engi- 
neering and maintainability function in support- 
ing the design/development process, the evalu- 
ator should check on the documented 
responsibility for this activity and how it op- 
erates; for example: 

(1) Has the contractor planned a human 
factor and maintainability program commen- 
surate with the overall requirements of the 
system? 

(2) Does the contractor have specialized 
personnel (in-house or otherwise) to carry out 
the proposed program successfully, and are 
their responsibilities specified in a project or 
management directive? Is there an adequate 
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mechanism for assuring effective consultation 
between designers and these specialists? 

(3) Does the designer coordinate early de- 
sign information with the human engineering 
specialists prior to design release for manu- 
facture ? This should be accomplished both 
informally and formally prior to the pre- 
packaging design review. 

(4) As design changes are made, are they 
coordinated with the human engineering special - 
lists (or functional equivalent) to assure proper 
consideration of maintenance and human fac- 
tors ? 

(5) Do those responsible for thehumanfae- 
tors effort regularly assess and report the 
status of actions being taken to cope with rec- 
ognized problems in the human factors area? 

(6) If no separate human engineering and 
maintainability specialists are available, how 
and by whom is this function being carried out ? 

(7) Are the steps of the program to elimi- 
nate human error sources actually being com- 
pleted in a satisfactory manner? 

The evaluator should carefully consider the 
needs of the project in question with regard to 
human factors support and seek expert opinion 
if necessary to identify effectiveness of activi- 
ties and the efficiency of use of manpower for 
this function. 

5.9 DESIGN REVIEW 
Contractor Requirements 

General 

A means must exist within the project 
whereby the design progress of eachsystem ele- 
ment can be periodically assessed by project 
management as the basis for confirming, re- 
orienting, or rescheduling project effort. The 
contractor is therefore required to establish a 
program of design reviews applicable at the 
major component, subsystem, and system level 
and scheduled so as to review technical de- 
cision and trade-off information at major proj- 
ect milestones. Similar reviews should also be 
required of all major subsystem and component 
suppliers , and these should be monitored by the 
contractor (see section entitled "Subcontr actor 
and Supplier Control." 


The design review activity is basically a 
survey of each hardware segment, in which all 
design requirements are reviewed (basic func- 
tions, input-output parameters, environment 
conditions, reliability, maintainability, weight, 
volume requirements, etc.) and assurance is 
obtained for company and customer management 
that the design has been optimized to support 
these requirements. A complete design review 
should consist of (1) the necessary pre- meeting 
compilation of all design considerations (i.e., 
the design review "data package"), (2) the de- 
sign review meeting, and (3) a formal report of 
all important results of the meeting. Project 
action items should be generated as a result of 
each review and each item should be assigned to 
a responsible group on the basis of priority of 
importance to the project. 

This subject is treated at some length in an 
as yet unpublished paper entitled "Elements of 
Design Review for Space Systems. " The five 
principal design review milestones described 
therein are: 

(1) Preliminary: at the completion of ini- 
tial formulation of a design concept, where it 
can be expected that several approaches to the 
design are feasible, and where circuit configu- 
ration decisions offer varied choices. 

(2) Prepackaging: between the preliminary 
and prerelease reviews , where an interim re- 
view of critical parts and circuit selection and 
application can be of definite benefit. 

(3) Prerelease: prior to release of the 
drawing and specification package to manu- 
facturing (breadboard and early development 
test results are available). 

(4) Postqualification: after completion of 
component qualification but prior to delivery of 
the qualification report. The purpose is to as- 
sure that all changes to the qualification units 
are incorporated in the delivered hardware and 
that all testing to date confirms the adequacy of 
the design including any revisions since design 
release. 

(5) Acceptance review: a part of the buy- 
off procedure prior to government acceptance 
of the hardware to give assurance to the cus- 
tomer that the system is in accordance with 
specifications and mission requirements. 
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Effective design review is contingent upon 

(1) the delegation of review responsibility to 
personnel having both high technical qualifica- 
tions and the ability to communicate their ideas 
effectively and (2) sufficient preparation prior 
to the review by all persons involved. All rep- 
resentatives must have a complete understand- 
ing of the requirements of the hardware under 
review that are applicable to their discipline. 
By the time of the review meeting, each should 
have studied the design review "input data 
package" and have prepared questions on this 
package and on the results of all previous in- 
formal reviews and critiques of individual seg- 
ments of the hardware. Although the design re- 
view requires that the designer defend his 
hardware configuration, it should be conducted 
with sufficient objectivity that the designer is 
not placed "on trial" or in a position of personal 
defense. 

The procedure for conducting the design re- 
view meeting may be oriented by a checklist or 
tabulation of pertinent items to be considered. 
However, whether a checklist is, or is not, 
used, the important point is that the meeting 
proceedings should be oriented to the following: 

(1) The hardware design specification 

(2) The latest mission profile requirements 
for the subsystem being reviewed and other re- 
lated subsystems 

(3) Questions on the input data package 
generated by each member of the design review 
team prior to convening the meeting. 

The result of each design review should be 
a report that completely describes the design 
status at the time of review, documents the re- 
view discussion, explains the designer’s means 
of meeting each requirement, lists any areas of 
nonconformance, discusses solutions to provide 
conformance, and points out trade-off consider- 
ations that would be involved in decisions to 
change the design. The design review report is 
the principal input for assurance to company 
and customer management that previously made 
technical decisions are valid, and it may rec- 
ommend action required to achieve desired de- 
sign improvements. 

Reliability Group Responsibilities 

As a support function to the project, the 
reliability group should actively participate in 


all design reviews from two standpoints. The 
first of these is to determine whether the design 
has completely responded to previously es- 
tablished reliability requirements and, if not, 
whether they have been given proper consider- 
ation in the trade-off decisions. The reliability 
representative should also assure that any pre- 
vious reliability group recommendations (as a 
result of earlier reliability prediction or analy- 
sis) have been duly considered and implemented 
where feasible within other trade-off limita- 
tions . 

The second reliability group function is to 
monitor the overall design review activity to 
ensure that adequate documentation of all prob- 
lem areas is achieved, to follow up on action 
items, and to advise the chairman of the design 
review team (or project manager) of status and 
thoroughness of implementation of these follow- 
through activities. This support to the design 
review chairman (or project manager) falls in 
naturally with the reliability group’s engineer- 
ing monitoring activities overall. 

Evaluation Considerations 

NASA’s concern with the contractor’s de- 
sign review program is much broader than the 
single aspect of evaluation and monitoring of 
reliability assurance with which this text is pri- 
marily concerned. Because the design review 
provides mid-program summations and analyses 
of design progress at significant program mile- 
stones, the NASA project management team 
should be acutely concerned with the review 
program. This concern will normally include 
close monitoring by NASA design and project 
engineering personnel of any hardware items 
within their individual areas of technical re- 
sponsibility. 

The reliability program evaluators should 
therefore coordinate with their design and proj- 
ect engineering counterparts and other support- 
ing engineers in obtaining an overall assurance 
that the design review program is being imple- 
mented satisfactorily. In addition, to complete 
the evaluation, they should perform an inde- 
pendent assessment of the contractor reliability 
group’s participation and of management’s use 
of the design review outputs. From this latter 
standpoint, they should review the project de- 
sign review planning documentation and ascer- 
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tain that a design review program has been es- 
tablished which provides for reviews of all 
system elements down to the major component 
level, that the program includes subcontractor’s 
design review activities, and that it adequately 
defines the participants and their responsibili- 
ties. Then the contractor reliability group par- 
ticipation can be evaluated by the checking of 
such details as the following: 

(1) Does the reliability group have repre- 
sentation in all design review meetings? 

(2) Is the reliability group supplying up-to- 
date reliability predictions and FMECA (see two 
previous sections) as part of "the input data 
package" ? 

(3) Is there a mechanism for assuring 
thoroughness of review documentation and 
follow-through on action items ? If not other- 
wise assigned to one specific group, are these 
functions assigned to reliability? 

(4) Is a reliability representative among 
those signing off on the final report, signifying 
concurrence in its content? 

(5) Are the right personnel in attendance at 
each design review? 

A vital factor is the importance which the 
contractor’s top project management attaches 
to the design review program. In this area, the 
evaluation should concern itself with the answers 
to questions such as the following: 

(1) To what extent is project management 
actively participating in the design reviews, and 
at what level (of personnel) does project man- 
agement show an immediate interest in specific 
design review activities? 

(2) Are design review outputs being em- 
ployed as basic project guides, and is manage- 
ment issuing timely direction for action as a 
result of the reviews ? 

(3) How is the responsibility for monitor- 
ing subcontractor design reviews delegated? 
Are contractor representatives actually par- 
ticipating in these reviews and concurring in 
the resultant final reports? Where deficiencies 
are considered to exist, does the contractor 
project management have a means (and is he 
employing it) for frequent checkup on subcon- 
tractor progress to bring about a satisfactory 


solution? Such a means might be a contractor 
resident engineering group, the contractor 
quality assurance representative, or other con- 
tractor representatives reporting frequently to 
the project manager. 

(4) Are subcontractor design review re- 
ports provided in accordance with project sched- 
uling, and are they sufficiently detailed to 
assess the comprehensiveness of the review? 

Since the results of other reliability as- 
surance functions (predictions, FMECA, de- 
sign standardization, parts and materials stand- 
ardization, etc.) are all keyed to, or have a 
bearing on, the design review, this function 
provides a focal point for the evaluator at which 
he may assess not only that adequate design re- 
views are being conducted but that the outputs 
of these supporting reliability tasks are being 
furnished in a timely fashion; moreover, he may 
assess the extent to which they are impacting 
the project. For these reasons, the design re- 
view activity should be given top priority in the 
overall reliability program assessment. 

5.10 FAILURE REPORTING AND CORRECTION 

Contractor Requirements 

The contractor is required to have a for- 
malized system for the reporting of hardware 
(and software) discrepancies and nonconform- 
ance to design specifications. This system 
serves to document equipment discrepancies 
not only in quality and workmanship but also in 
performance of all system elements at various 
levels of testing. While data on the former type 
of discrepancy give a good indication of the 
manufacturer’s capability to produce a quality 
product (and can serve as a check on the quality 
control operation), data on the latter are im- 
portant to both the measurement and assurance 
of system reliability. 

The primary purpose of the failure report- 
ing system is to document accurately all dis- 
crepancies and to disseminate this information 
so that remedial action may be taken promptly 
and thoroughly, thereby preventing recurrence 
of the failure or anomalous condition. To serve 
this purpose adequately, the contractor must 
maintain a disciplined reporting system and an 
effectively managed corrective action scheme 
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with sufficient followup to assure that the cor- 
rective actions are taken and that the problem 
condition is eliminated. 

The reporting system not only must exist in 
the contractor’s main plant, the subcontractor s' 
plants, and all hardware test facilities (includ- 
ing the launch facilities), but it must also func- 
tion at command and tracking facilities as- 
sociated with the mission operations system; 
that is, it must encompass people, procedures, 
equipment, and communications associated with 
the data acquisition, tracking, command, and 
mission operational control of the launch vehicle 
and spacecraft. In cases where the contractor 
does not have responsibility for mission opera- 
tions, the cognizant NASA center must assume 
the contractor’s failure reporting assurance 
responsibilities in this area and coordinate this 
activity with that of the contractor. 

In the subcontractor facilities and in the 
mission operations facilities, these organiza- 
tions must be required to maintain a failure 
system, which, if not implemented with the 
identical paperwork format, at least provides 
for the recording of all failure information nec- 
essary to evaluate each hardware or software 
discrepancy effectively. 

Because of the remoteness of some of these 
locations from the contractor’s plant and the 
large volume of data that results from all loca- 
tions, it is necessary that the contractor desig- 
nate a group to be responsible for: (1) As- 
sembling all failure reports, (2) determining 
the criticality of each failure and the priority 
for failure analysis, and (3) upon completion of 
such analysis, monitoring the progress of cor- 
rective actions. The group discharging this re- 
sponsibility (frequently reliability or quality) 
should utilize the technical assistance, as re- 
quired, of particular design or systems engi- 
neers. A description of a typical failure re- 
porting system is included as a supplement to 
this text. 

There are several ’’keys” that render the 
failure reporting and corrective action cycle 
effective. Some of these are: 

(1) The discipline of the report writing it- 
self must be maintained so that an accurate de- 
scription of failure occurrence and proper 
identification of the failed items are assured. 


(2) The proper assignment of priority and 
the decision for failure analysis must be made 
with the aid of cognizant design engineers and 
systems engineers. 

(3) The status of all failure analysis must 
be known; it is of prime importance that the 
analysis uc expcuiicu do ^ jx . n»y demands and 
that corrective action be implemented as soon 
as possible. 

(4) There must be a means of tabulating the 
failure information for determining failure 
trends and the mean times between failures of 
individual system elements. There should also 
be a means for management visibility into the 
status of failure-report dispositions and cor- 
rective actions . Such a tabulation system may 
be either manual or part of an automatic data 
processing (ADP) system. For sizeable vol- 
umes of data, ADP is most helpful and should 
be utilized if the contractor has the capacity 
available . 

(5) Lastly, an extremely valuable assur- 
ance mechanism is to provide for a high-level 
technical management sign-off, concurring in 
the results of failure analysis, the soundness of 
corrective action, and the completion of formal 
actions in the correction and recurrence pre- 
vention loop. 

Evaluation Considerations 

The aspects of performance quality and 
timeliness (as discussed in chapter 3) are par- 
ticularly appropriate to the failure reporting 
and corrective action system. The NASA eval- 
uation should be primarily concerned with the 
completeness of information contained in the 
failure reports and the time lag between the re- 
porting of a typical failure and the implementa- 
tion in hardware of a solution to the problem. 
The two significant items of information (the 
description of the trouble recorded at the time 
of failure and the information resulting from 
failure analysis) may be contained in a single 
document or in two separate documents. A 
number of systems exist for the actual record- 
ing process. A typical failure reporting format 
(employed by the Jet Propulsion Laboratory of 
the California Institute of Technology) with ex- 
planation of its use appears as a supplement to 
this text (pp. 51 to 67). 
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If a system of separate documents is in ef- 
fect, a cross-reference numbering system 
should also exist. From these the evaluator 
should be able to determine the extent of the 
detail used in failure reporting and the depth to 
which the contractor performs analyses of par- 
ticular failures . Instead of examining a random 
selection of failure reports , the evaluator should 
select several representative failures (major 
ones, minor ones, and some involving subcon- 
tractors in the loop); he should then check the 
paperwork, failure analysis, and corrective ac- 
tion through the complete closed loop, carefully 
noting periods of time required for action, ap- 
parent delays in communications, thoroughness 
of documentation, and any other evidence to as- 
certain whether sound system discipline and 
sound technical judgment characterize the im- 
plementation of the failure reporting system. 

Coordination with a resident project engi- 
neer will usually uncover several suitable fail- 
ure reporting examples to be put to the evalua- 
tion test . In following through each typical case , 
the evaluator should direct his investigation in 
part toward determining the following: 

(1) Is the failure properly described in the 
failure report ? Typical examples of poor nota- 
tion are "no output," "doesn ! t work," "shorted, " 
"broken, " "no response. " Preferable descrip- 
tion of a failure would be, for example, "Out- 
put voltage of 7.5 Vdc was recorded during step 
29 of test procedure A12456. Tolerance is 8.2 
to 9.6 V dc." 

(2) Is the proper failed assembly recorded? 
One typical mistake is to identify a part or com- 
ponent as having failed when the failed item was 
actually another part or component in series. 

(3) Who determines whether a formal fail- 
ure analysis should be performed ? This task 
should be performed by reliability engineers in 
conjunction with system engineers to assure that 
all potentially critical failures are analyzed. 

(4) What is the time lag from the date of 
failure to that of the performance of failure 
analysis ? Even in a well-executed program the 
time for analysis may be expected to vary with 
such factors as criticality of the failure, ex- 
tensiveness of the analysis required, and 
whether the analysis must be done by a vendor. 
However, other programs may suffer from a 


lagging analysis effort because of "logjams" in 
paperwork, improper assignment of priorities, 
inadequate (or overcrowded) facilities , or simply 
inept management. The evaluator should ex- 
amine these factors in judging what constitutes 
too great a lag and whether long delays are the 
rule or the exception. 

(5) Is an adequate failure analysis con- 
ducted? Adequacy is usually recognizable in 
the failure analysis report by the completeness 
and detail of the description of the testing per- 
formed to isolate the failure and by a clear 
identification of the individual failure mechanism 
and probable cause. Failure analysis labora- 
tories may also be visited for more information. 

(6) Are useful corrective recommendations 
made? Again, the failure analysis report should 
show this. The individual responsible for the 
analysis should coordinate informally with the 
appropriate design group to discuss feasible 
solutions . 

(7) What is the time lag from the date of 
recommendation to actual implementation of a 
design change? Such action should be taken on 
a priority basis, most critical failures being 
considered first. 

(8) Check failure reports from outlying 
areas such as remote test facilities. Is there 
swift communication of critical failures from 
these areas to the main plant? 

(9) Where launch site failures occur, is 
there a special means of reporting these back 
to the plant (e.g. , by teletype) and obtaining 
prompt support for failure analysis and cor- 
rective action? 

(10) Is the failure reporting system of each 
major subcontractor properly monitored by the 
contractor, and are all failures resolved in a 
timely fashion? 

(11) What provisions are there for report- 
ing, analysis, and corrective action in the mis- 
sion operations system (including test and train- 
ing) ? Who is responsible, and how , in detail, 
does he perform these functions and coordinate 
them with the overall failure reporting system ? 

(12) Where a failure occurs in testing at the 
subsystem or system level, what provision ex- 
ists to ensure that testing does not continue with 
the failed, but unanalyzed, black box still 
in the system? (I.e. , is the failed black box 
removed and is system testing suspended until 


RELIABILITY PROGRAM ELEMENTS AND THEIR EVALUATION 


43 


either a replacement unit is obtained or the 
failed unit is analyzed, repaired, proofed, and 
reinstalled?) 

(13) Are all critical failures reconciled 
prior to utilization of the system? 

(14) Is there accountability for failure re- 
port forms; that is, are the blank forms serial- 
ized in printing and periodically accounted for 
to provide a check on whether filled out reports 
are being turned in promptly or are left lying 
about ? 

(15) Does technical management review and 
concur in corrective actions for closures of re- 
ported failures? Is there single-point responsi- 
bility for following and certifying the completion 
of all actions necessary to close a failure re- 
port ? 

(16) At what point in the life of an element of 
hardware does it become subject to formal fail- 
ure reporting (i.e., at module or component 
level and at what phase of testing) ? If failure 
reporting does not begin at the earliest stage of 
life and test of an identifiable item, what other 
mechanism exists for reporting of anomalous 
performance of the item prior to formal failure 
reporting ? 

(17) Does the contractor maintain and pub- 
lish a rapid- reaction, continuously updated 
management report which summarizes the status 
of all open failure reports and provides full 
visibility on items such as date of initiation, a 
very brief description of failure, test where 
failure occurred, affected component, individual 
responsible for close-out action, and anticipated 
(or required) disposition date? Does this sys- 
tem cover failure reporting status for subsys- 
tems and components being tested (or already 
delivered) by major subcontractors? 

(18) What are the requirements for closure 
of a failure report ? Does closure require de- 
vising, test- verifying, and incorporating the 
"fix" into the immediate flight hardware, the 
drawings, and other items of hardware in the 
same configuration series ? 

Although there are other individual items 
worthy of checking in particular contractor op- 
erations, these questions form a good basis for 
the investigation of any failure reporting system 
and should be added to as the evaluation pro- 
ceeds. Once the contractor has committed a 
design to manufacture, the failure reporting and 


corrective action system is the primary means 
of documenting design or manufacturing prob- 
lems, and sufficient emphasis must be given to 
each step in the reporting cycle to assure reso- 
lution of all critical hardware and software 


5.11 STANDARDIZATION OF DESIGN PRACTICES 
Contractor Requirements 

Hardware reliability is considerably en- 
hanced when the design and manufacture are 
accomplished in accordance with a well docu- 
mented and implemented standards system. By 
contrast, the lack of standardization of such 
practices causes excessive individualism of de- 
sign and an increased burden to project support 
and checking functions (and management) ; in most 
instances, this results in a lack of firm control 
of the hardware development. Although the de- 
velopment of unique design and manufacturing 
techniques should not be discouraged, a system 
must exist by which such development may be 
controlled and the new methods tested and 
proved before becoming acceptable within the 
project. 

The contractor will normally have a stand- 
ards program in operation prior to contracting 
for the NASA project. Every effort should be 
made to utilize as much as possible of this ex- 
isting standards program since such practice 
makes for better centralization of control within 
the contractor's facility and is an obvious cost 
saver. However, it is to be expected that these 
existing standards will reflect levels of refine- 
ment and process control commensurate with 
the type of hardware activity in which the con- 
tractor has been e ngaged. Therefore, in adapt- 
ing to an R&D space system effort, it will be 
necessary to review these standards and prac- 
tices to determine which among them require 
upgrading. This review should be the joint re- 
sponsibility of the contractor's reliability and 
quality assurance personnel and should result in 
a program for upgrading by the standards group. 

The program of standard practices should 
include documentation of manufacturing process 
specifications; general design techniques for 
electrical, electronic, and mechanical applica- 
tions; drafting guidelines; and installation tech- 
niques. Standardization of parts is also re- 
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quired; this will be discussed in the following 
section. The control of standard practices 
should be administered by a group specifically 
designated for this function. 

Although there are obvious reliability im- 
plications imposed by standard practices, con- 
tractor quality assurance personnel must also 
monitor the generation of new practices and take 
the leading role in monitoring the implementa- 
tion of standard techniques , particularly manu- 
facturing processes, to assure that quality as- 
surance requirements in the areas of process 
control procedures and environments are met. 
(See par. 7.5.4 of NASA quality publication 
NPC 200-2 (ref. 7).) This should include their 
being in the concurrence signature cycle of all 
standard procedures documentation. In the area 
of standard design practices, the concurrence of 
appropriate assurance groups should also be 
required . 

Evaluation Considerations 

The evaluation of the contractor’s standards 
area should aim at the adequacy of the standards 
themselves and at the proper use of the stand- 
ards in design and manufacturing. This in- 
cludes determination of: 

(1) Whether the process specifications be- 
ing used represent acceptable practices and 
provide adequate controls in terms of needs of 
the space system in question 

(2) Whether available standards are being 
properly utilized through drawing and specifi- 
cation callouts 

(3) Whether the contractor’s quality pro- 
gram is adequately enforcing proper imple- 
mentation of manufacturing process controls. 

Item (1) above requires an independent re- 
view of each standard process and practice with 
a view toward ease of manufacture, ability to 
control the process, and adequacy of contractor 
facilities . Item (2) can be evaluated through a 
periodic review of samples of drawings and 
specifications by either residents or other task 
team delegates and by giving this area con- 
sideration in design reviews. 

Because of the importance of assuring that 
implementation of process controls is main- 
tained at all times, item (3), a customer- 


monitoring effort is frequently appropriate to 
assure that the contractor’s quality assurance 
function is performing adequately. Government 
agency representatives, if available, should be 
utilized for this task, and provisions for their 
employment for it is covered in NASA quality 
publication 200-1A (ref. 6). In this function, 
they should establish a schedule of monitoring 
to be carried out throughout the duration of the 
project. Their surveillance should include 
checking of control measurements performed 
during the process, review of all instrumenta- 
tion calibrations, and a determination that con- 
tractor quality control personnel are maintain- 
ing tight control of each process in all other 
appropriate ways . 

Other considerations for evaluation of the 
contractor reliability and quality assurance 
group’s tie-in to the contractor standards ef- 
fort are implied in the preceding section and 
are briefly restated as follows: 

(1) Is there evidence of a planned, sched- 
uled effort by contractor R&QA for early review 
of existing standard processes for applicability 
to the present space-system project? 

(2) Has an orderly program of updating of 
process specifications been implemented? 

(3) Is reliability assurance in the concur- 
rence signature cycle for design practices ? 

(4) Is quality assurance in the concurrence 
signature cycle for all standard manufacturing 
processes? 

(5) Does the contractor’s quality assurance 
group execute a planned, periodic program of 
checking process implementation on the manu- 
facturing floor ? 

5.12 PARTS AND MATERIALS PROGRAM 
Contractor Requirements 

In order to furnish support for the design 
and development effort, the contractor must 
conduct a program to control the selection and 
application of parts and materials . This program 
is one of the foundations of the development of 
reliable systems; as such, it must be system- 
atically implemented with the respect its impor- 
tance demands . 

The contractor must establish a group to 
manage and execute this program as required 
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by the project. Functions should include select- 
ing, specifying (i.e., providing specifications), 
and qualifying parts and materials to result in a 
parts list for the project. This list must be 
maintained current with respect to parts avail- 
ability and new project or mission requirements. 
This parts and materials group should provide 
consultation to design engineers to guide them 
in selecting parts appropriate for each appli- 
cation in terms of system performance and 
environmental requirements (utilizing suitable 
derating criteria and techniques). The parts 
group should also support the failure analysis 
activity in the analysis of failed parts. The 
reliability group should monitor the parts and 
materials function to ascertain whether it is 
supporting the project effectively in these areas. 

It is important that the parts group be con- 
sulted by design engineers as early as possible 
during the formulation of basic subsystem de- 
signs so that applications for which no standard 
part exists can be identified in time to per- 
mit consideration of minor design changes 
which may eliminate the need for procurement 
of nonstandard items. When a nonstandard part 
must be used, adequate specifications must be 
drafted early to avoid delays in the procurement 
cycle. Consultation with parts specialists should 
be continued throughout the development period 
whenever additional parts application problems 
arise. The parts and materials group should 
sign off on design drawings to indicate their 
review of parts applications. Also, parts and 
materials application reviews should be con- 
ducted and documented as inputs to design 
reviews of the components and subsystems in 
question. (A detailed treatment of this review 
area is given in an as yet not published docu- 
ment entitled "Parts and Materials Application 
Review for Space Systems.") 

Normally, the parts program will develop a 
preliminary parts list early in the project cycle, 
which will evolve through qualification and test- 
ing into an approved parts list (APL) as the final 
parts reference document for the design. The 
APL must include specifications as to which each 
part is qualified, the recommended application 
guidance for the part , and pertinent warnings of 
limitations on its usage. As additional parts are 
added to the parts list through qualification, 
these parts must be included on updated ver- 


sions or in supplemental pages of the approved 
parts list. 

The parts and materials group has the two- 
fold responsibility of continuously monitoring 
progress in the industry in order to maintain 
up-to-date standard information for dissemina- 
tion to the project and of consulting with project 
designers on the selection and application of 
parts and materials. 

Evaluation Considerations 

The above description of the contractor’s 
requirements cites some of the principal factors 
to be considered in the evaluation. The organ- 
ization of the parts and materials activity, as 
outlined in the contractor’s planning documenta- 
tion, should be reviewed by NASA center per- 
sonnel. It should be determined that a separate, 
full-time responsibility has been delegated for 
the purpose of performing the functions cited in 
the preceding section. The official mission 
statement of the group should indicate an ap- 
proval authority for all parts and materials 
selection, and the detailed schedules of the 
project should identify points in time for parts 
and materials applications review of each com- 
ponent design. 

Detailed implementation of the parts and 
materials activity should be monitored by NASA 
residents and/or by the delegated survey group 
during periodic surveys of the reliability pro- 
gram operation. Some of the specific items to 
check are the following: 

(1) To what extent do designers solicit parts 
and materials assistance from the parts group? 

(2) Are the personnel in the parts group 
qualified ? Interview them on an informal basis 
to determine whether the collective experience 
of the group encompasses electronic, electrical, 
electromechanical, and mechanical parts knowl- 
edge. 

(3) Do the design personnel appear to re- 
spect the technical ability of this group? (De- 
termined from evaluator's discussions with 
design personnel.) 

(4) Does the parts and materials group 
make review of parts applications as a specific 
input to the design reviews of each component ? 

(5) Is there a preliminary parts list for the 
project? If the "drawing release" stage has been 
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reached, is there an APL, and is it under proj- 
ect drawing change control?" 

(6) Is the parts group responsible for as- 
sisting in the specification of requirements for 
all parts (nonstandard and standard) , and is there 
an accessible listing of these parts for possible 
use in other applications ? 

(7) Does the parts and materials group have 
approval authority over sources from which 
parts and materials are purchased? 

(8) What means are used to ensure that 
parts for use on flight-critical hardware are 
obtained from approved vendors ? 

(9) Are any parts purchased from sources 
other than approved vendors? If so, for what 
use are they intended, and how are they con- 
trolled ? 

(10) What controls exist to ensure that any 
short- order system, by which designers pur- 
chase limited numbers of parts for breadboard 
or other informal testing, does not degenerate 
to side purchase of critical components for the 
sake of expediency? 

(11) For projects where the formal failure 
reporting system does not cover breadboard or 
other testing prior to flight acceptance testing 
(FAT), what mechanism is used by the parts 
group to keep track of catastrophic part failures 
in early testing ? 

5.13 EQUIPMENT LOGS 

As the development cycle progresses and 
the number and types of equipments of various 
configurations increase, the problem of hard- 
ware configuration control takes on major 
significance. The contractor must have a means 
of document ing the history of every manufactured 
item and must enforce discipline in all plant 
facilities to keep such documentation up to date 
at all times. Although this task is a natural 
requirement in the area of test and product 
quality control, it directly affects system re- 
liability. Questions arise daily (particularly in 
test situations) concerning the latest incorpo- 
rated changes to individual units, their most 
recent performance history, previous discrep- 
ancies, and conformance to specifications. This 
information must be readily accessible and ac- 
curately listed in an equipment log to allow for 
efficient test operations as the equipment pro- 


gresses from one area to another and to provide 
a detailed record to support later failure 
analyses . 

Section 3.10 of NPC 250-1 (ref. 1) provides 
a detailed listing of the items that should be 
included in the equipment log, and paragraph 
14.2.4 of NPC 200-2 (ref. 7) prescribes addi- 
tional logging requirements; these need not be 
repeated in this writing. Evaluation by the NASA 
contract monitoring team is also relatively 
straightforward since the referenced listings 
provide a detailed itemization of information of 
concern. The evaluation can only be performed 
practically as a matter of routine along with 
other daily duties, rather than by a wholesale 
review of all logs at one time . 

To meet this requirement, the contractor 
may elect to record parts of the total data com- 
pilation in several documents rather than in a 
single log. Test results (in particular those 
recorded in real time by oscillograph or pen 
recorders) may only be referenced in the equip- 
ment log to another data file. Such a technique 
is satisfactory as long as all the data and refer- 
ences are recorded currently and are always 
readily available in a repository reasonably near 
the equipment. 

The equipment log data are not only impor- 
tant to the efficient progress of the test activities 
but they become one of the key documents in the 
customer-contractor negotiation of system ac- 
ceptance. Where discipline in recording the 
required data has not been maintained, the de- 
ficiency will become apparent and constitute an 
obstacle to confidence in the product at system 
acceptance. It is therefore important for the 
NASA reliability program evaluation to monitor 
the equipment logs for adequacy beginning as 
early as practical in the development phase. 

5.14 TESTING AND RELIABILITY EVALUATION 
Contractor Requirements 

Once system development progresses to a 
stage where hardware is being produced, the 
emphasis of the reliability program must be on 
contractor activities that provide data for meas- 
urement and assessment of actual hardware 
capability. It is during this period that the 
analytical reliability engineering efforts are 
provided support in actual data, and previous 
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analyses can be refined as a result of testing of 
the system at various levels of assembly. 

In most projects, the size, scope, and 
technical skill levels required to carry out a 
successful test program dictate its being planned 
and conducted by a separate organization. How- 
ever, it is the responsibility of reliability man- 
agement to accomplish the following: 

(1) Determine and require the test data 
necessary to support the analytical reliability 
assurance tasks . 

(2) Assist in the planning of the test pro- 
gram. 

(3) Provide (in conjunction with the quality 
assurance group) a monitoring effort to assure 
that the test program supplies the basic infor- 
mation required and that the data have been 
recorded under the conditions prescribed by 
applicable documentation. 

The contractor must therefore provide in 
his reliability program plan a detailed descrip- 
tion and schedule of the test activity as it inter- 
faces with the reliability assurance program 
(reliability evaluation plan) and show how the 
data generated by the tests will be used to sup- 
port the analytical reliability evaluation process. 

In regard to the test monitoring function, 
the contractor reliability group, together with 
the quality assurance group, must closely follow 
the test program to assure that all testing is 
adequately performed and documented. This 
process should include review of test specifi- 
cations and test procedures for adequate content, 
monitoring of the test implementation, and re- 
view of test reports. The reliability group 
should be actively engaged in the evaluation of 
all failures resulting from testing and should 
assure that adequate documentation of all oper- 
ating time , cycles , and anomalous conditions is 
maintained (see preceding section). Further, 
the reliability group should compile all data on 
failure versus operating time to refine hardware 
failure rates and to verify previous system re- 
liability predictions. 

As part of the overall test plan, the 
contractor should perform reliability tests at 
the part, selected-component, and sometimes 
even higher levels. However, because of Ihe 
cost of such testing on a large scale, the extent 


to which it is carried out should be established 
in contract negotiations. Factors cited in the 
section entitled "Response to Project Charac- 
teristics" in chapter 2 such as mission criticality, 
type, and cost have a direct bearing on the 
scope of reliability testing to be conducted on 
parts and components. 

Additional verification of the capability of 
hardware to perform reliably is obtained through 
test results which: 

(1) Verify performance capability at the 
component, subsystem, and system level 

(2) Identify undefined subsystem interfaces 

(3) Identify design weakness and defects in 
materials or workmanship 

(4) Verify the performance of hardware 
when it is subjected to expected mission environ- 
ments and durations 

(5) Assure the required life (accelerated 
when necessary) of all critical system compo- 
nents 

(6) Apply overstress to determine margins 
of safety of all components and subsystems 

Each of the above segments of testing pro- 
vides data which either verify adequate perform- 
ance or identify areas requiring improvement, 
thus providing additional confidence in ultimate 
mission success. 

Evaluation Criteria 

The test program of any project interfaces 
with almost every group and discipline within 
the project. It is performed not only for the 
purpose of providing inputs to the reliability 
program but also to demonstrate design integrity, 
to prove the design is capable of qualifying to 
all mission environments, to determine com- 
patibility of one subsystem with another, and to 
demonstrate readiness for acceptance by the 
customer. The information resulting from test- 
ing is therefore of vital interest to all NASA 
project personnel. 

Although a significant NASA test program 
monitoring effort will, in most cases, be taking 
place in addition to the reliability program eval- 
uation effort, the latter evaluation should focus 
upon the interface between the reliability pro- 
gram and the test program. The preceding sec- 
tion implies the major considerations to be de- 
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termined from the standpoint of planning an 
integrated activity and providing the necessary 
reliability support data. Further questions to 
be answered are: 

(1) Does the RPP adequately define the 
interface between the reliability program and 
the test program and specify the test data that 
must be provided for reliability assurance ? 

(2) Has the reliability group (in coordina- 
tion with quality assurance) established a plan 
for test monitoring, and is the plan being 
implemented in an effective manner ? 

(3) Are all test procedures being reviewed 
and compared with the corresponding test speci- 
fications, and are inconsistencies discovered in 
this review being reconciled by feedback to the 
test group ? 

(4) Does the reliability group maintain a 
disciplined collection program for operating time 
and cycle data from all test areas ? 

(5) Are component failure rates updated by 
analysis of testing time versus failure data? 
Are these failure rates periodically applied in 
revising reliability predictions to obtain refined 
reliability assessments ? 

(6) Is the reliability test program for parts 
and components being implemented as estab- 
lished by contract, and is it progressing satis- 
factorily ? 

(7) Does the contractor have an organized 
process for maintaining control of failed items , 
and analysis of failure, and the followup for 
corrective action? Is this process meticulously 
carried out (see section entitled "Failure Re- 
porting and Correction") ? 

(8) Is the contractor employing statistical 
test planning techniques to achieve maximum 
economy in testing ? 

(9) Is the reliability evaluation plan sub- 
mitted initially as apart of (or at the same time 
as) the formal RPP ? 

(10) Does the reliability evaluation plan 
clearly show (preferably in matrix or other 
graphic form) tests conducted at each level of 
assembly which contribute to reliability assess- 
ments and does it show 

(a) What each test contributes , with relative 
value of the contribution to the as- 
sessment 


(b) Program milestones at which reliability 

data from each test will be available 

(c) Estimate of cost of additional effort in 

each test toproduce specific reliabil- 
ity data 

While the reliability program monitoring 
and evaluation is proceeding, much of the day- 
to-day evaluation of the test program will usually 
be carried out by NASA project engineers be- 
cause of the detailed technical decisions re- 
quired between customer and contractor in this 
activity. However, the monitoring of testing to 
assure compliance with procedures, particu- 
larly at the component or subsystem level, will 
usually be delegated to government agency 
representatives (when available) . 

NASA reliability monitoring personnel and 
project engineers should be on the alert for 
indications of a poor test program. If the test 
program is not producing the necessary data for 
reliability assurance or the many other decision- 
making criteria that it must provide, a test pro- 
gram evaluation should be conducted to determine 
the weaknesses and how to correct them. This 
effort might include a study of all specifications, 
procedures, test implementation, test item con- 
figuration control, and review of test results. 
Consideration may be given to employing a 
consultant group to assist in this study (see 
section entitled "Reliability Study Contracts" in 
chapter 4). 

5.15 RELIABILITY PROGRAM DOCUMENTATION 

Much has been said on the subject of pro- 
ducing sufficient documentation to record all the 
significant inputs to, and outputs from, the re- 
liability support effort. While it is true that a 
certain level of documentation is necessary to 
provide a basis for a well-managed project and 
as evidence of an adequately implemented pro- 
gram (in all areas), stated requirements for this 
should not be misconstrued as a requirement or 
encouragement for creating an excessive volume 
of paperwork as evidence of an adequately docu- 
mented program. What must result from the 
contractor's efforts are documents that either 
satisfy an internal project purpose or are con- 
sidered basic to the contractor-customer inter- 
face for proper management of the program. In 
this latter category are documents such as pro- 
gress, management, and financial reports, and 
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documents that provide direction to the project. 
All other documentation should have as a first 
purpose the dissemination of information, the 
recording of data necessary to the accomplish- 
ment of the project effort, or the documentation 
of underlying logic governing project decisions. 
Although internal project documentation must be 
available for review by the customer as deemed 
necessary to his overall monitoring activity, the 
generation of such technical documentation solely 
for the enlightenment of the customer is not 
appropriate . 

It is particularly important to avoid dupli- 
cations in reporting routine. Excessive docu- 


mentation is self-defeating in that it tends to 
bring about first a confused, then an indifferent, 
attitude toward all documentation. It is also 
recognized as a typical means of masking un- 
favorable technical information amid an array 
of irrelevant documents. 

Both the contractor and the customer evalu- 
ation activity should be conducted with an 
awareness of the primary purpose of the docu- 
mentation effort. Beyond the recognized need 
to provide information for management decision, 
the documentation must provide technical sup- 
port to the design, evolution, and use of hard- 
ware and to the accomplishment of mission. 
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Supplement— Spacecraft Problem/ Failure Reporting System 


The following exhibit describes the failure reporting procedure used by the 
Jet Propulsion Laboratory (JPL) of the California Institute of Technology. It 
has been selected as typifying most of the key features of the better failure re- 
porting systems presently in use in the aerospace industry. 

Most of the elements of this procedure, although tailored to the organizational 
structure of JPL, may be readily recognized as generic in their essence. How- 
ever, in reading paragraph IB entitled "Scope," it should be borne in mind that 
the lowest levels of assembly and types of tests, the failures of which are to be 
covered by the reporting system, should be interpreted separately for each proj- 
ect to conform with the lowest levels at which (1) there exists a full formalized 
characterization of test conditions, procedures, and stress levels and (2) a for- 
malized system of configuration control is in force. 
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DEFINITIONS 


ECR Engineering Change Requirement 


FAILURE Equipment performance outside the limits of 

applicable requirements. Term to include 
intermittent, cessation of performance, or 
failure of equipment to respond as commanded. 

p/FR Problem Failure Report 

PROBLEM Any anomaly requiring action for correction, 

or an occurrence which cannot immediately be 
explained. 

PTM Proof Test Model 


SAF Spacecraft Assembly Facility, Jet Propulsion 

Laboratory, Building 179 
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I. INTRODUCTION 


A. Objective 

The objective of this Problem/Failure Report (p/FR) Procedure 
is to describe a method of systematically reporting, recording, 
and reviewing all actions upon failures and problems that arise 
in the design, fabrication, test, and operation of spacecraft 
and related Operational Support Equipment. 

B. Scope 

This procedure applies to all JFL personnel. It is utilized 
in reporting problems or failures on Lab or at the Eastern Test 
Range. (p/FR procedures for JPL Contractors are contained in 
Reliability Procedure RP-2A). The Problem/ Failure Report shall 
be used to record all failures and problems occurring during 
Type Approval and Flight Acceptance tests of PTM, flight systems 
and flight spares and any other testing of subassemblies, 
assemblies, subsystems, and systems, commencing with the first 
application of power. 

II. RESPONSIBILITIES 

The items listed below are those necessary to implement the JPL 
Problem/Failure Reporting System. 

A. Personnel 

The roles of the various personnel in the JPL in-house p/FR 
Reporting System are as follows : 

1. Originator - The originator is anyone who believes a 
problem or failure has occurred or exists. 

2. Processor - The JPL Reliability p/FR Group (Section 153). 

3. Cognizant Engineer - That person who is responsible for the 
design and/or fabrication of an assembly or subsystem. 

4. Cognizant Section Chief (Manager ) - That person who is 
Section Manager for one or more cognizant engineers. 

5* Reliability Coordinator - Hiat person representing 
Section 153 on initiation, distribution, action, and 
closeout of p/FR activities. 
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6. Spacecraft System Engineer - That person responsible for 
the overall technical design of the spacecraft. 

7* Reliability Project Representative ‘or Reliability Manager - 
That person from the Reliability portion of Division 15 
assigned to each specific project. 

B. Applicable Documents 

JPL Problem/Failure Report (p/FR) Form, JPL No. 1290, 

January 1964. 

JPL Procedure RP-2A, JPL Contractor Problem/Failure Report- 
ing, dated 1 April 1966. 


SUPPLEMENT-SPACECRAFT PROBLEM/FAILURE REPORTING SYSTEM 


59 




Procedure No. RP-1A 

III. 
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OPERATION 


Origination 

NOTE: BE SURE TO PULL PROTECTIVE TISSUE WHEN FILLING OUT A P/FR 

AND REPLACE SAME WHEN DONE. 

A. The originator completely Tills out Section I of the Problem 
Failure Report (JPL-1290) or a facsimile thereof (Appendices A 
and B give examples of how to properly fill out. a p/FR) and gives 
it to: 

1. The cognizant Quality Assurance (QA) representative for 
handling (if the p/FR originates at SAF, Building 144, 
Building 150, or the Eastern Test Range), or 

2. The Cognizant Engineer for handling (if the p/FR originates 
from a bench test as a subsystem), or 

3. The Reliability p/FR Group* 

NOTE : IN THE EVENT TWO OR MORE SUBSYSTEMS ARE INVOLVED, THE SPACE- 

CRAFT SYSTEM ENGINEER IS NAMED AS THE COGNIZANT ENGINEER. 

B. Next, in the case of 1 and 2 above, the entire P/FR (the ditto - 
master and colored copies), except for the hard copy, is sent 
to the Reliability P/FR Group, Section 153* IT a failure, the 
hard copy remains with the failed part or assembly. If a 
problem, the hard copy remains with the Cognizant Engineer until 
the problem is analyzed, in case a failure is discovered during 
problem analysis. 

JPL 1290 Form Distribution (initial ) 

A. Distribution of the Problem Failure Reports will be handled by 
the Reliability p/FR Group. (Quality Assurance may perform 
this service at the Eastern Test Range). 

Routing by the Reliability p/FR Group will be as follows : 

1. The DITTOMASTER will be kept on file by the Reliability P/FR 
Group and copies of "open” p/FR's will be sent to appropriate 
personnel. Lists of such personnel will be supplied to the 
Reliability P/FR Group by the Reliability Project Repre- 
sentative. 

2. The PINK copy will be sent to the Cognizant Engineer whose 
name appears in the lower right hand corner of Block 9 on 
the P/FR form. 

3. The GREEN copy will be sent to the Spacecraft System Engineer. 
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STEP OPERATION 


4. The HARD COPY (cardboard copy) will be kept with the appropriate 
Problem/Failure hardware. 

The Reliability p/FR Group will log in the dittomaster and, if 
necessary, will seek any information missing from Section I needed 
to describe the problem or failure. 

3 Verification, Analysis and Corrective Action 

A. The Cognizant Engineer receives the Pink Copy and proceeds to in- 
vestigate and analyze the problera/f ailure . His analysis and re- 
commended action are recorded in Sections II and III. Ihe cogni- 
zant engineer should complete all data on the pink copy left out 
by the originator such as Blocks 4 a, B, C, or D. He should correct 
any errors that the originator may have made in Section I. The 
Cognizant Engineer makes the determination whether the occurrence 
is a problem or failure. If any further explanations for filling 
out these sections are needed, the Reliability P/FR Group or the 
Reliability Project Representative should be contacted. 

B. If a failed electronic or electromechanical part is involved, the 
Cognizant Engineer dispositions the part(s) to the Electronic Parts 
Engineering Section (Section 354) along with the hard copy. If the 
failed part(s) cannot be determined or removed from the subassembly 
without damage. Section 354 shall be contacted to determine the 
method to be used for the part's analysis. The cognizant engineer 
should retain the pink copy until the Parts Failure Analysis Report 
has been received from the Electronic Parts Engineering Section. 

C. If an ECR is required, the pink copy should not be forwarded to the 
Reliability p/FR Group until an ECR number has been assigned and 
effectivity established. 

D. The p/FR must be filled out such that it is self-explanatory and 
self-sufficient. Significant details should be reported in the 
analysis and corrective action blocks. Addenda and references 

may be used. The degree to which the analysis was performed should 
be indicated. The p/FR must contain sufficient information to 
allow an evaluation to be made of the problem, and its risk to the 
mission. 

E. If special tests are required to verify a failure or authenticate 
the adequacy of corrective action, the p/FR should be left open 
until the test work has been completed to the satisfaction of the 
Cognizant and the Spacecraft System Engineers. 


F. When all of the above has been completed satisfactorily, the Cogni- 
zant Engineer obtains his Section Manager's approval of his analysis 
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OPERATION 


and corrective actions and returns the completed pink copy to the 
Reliability P/FR Group as soon as possible after the initial distri- 
bution of the p/FR. The Section Manager signs for: (a) completeness, 

(b) adequacy, (c) definitive description, and (d) technical soundness 
of the analysis and corrective action defined on the p/FR. 

* 

Parts Failure Analysis 

A. The hard copy or reasonable facsimile is kept with the lowest level 
of hardware responsible for a problem/failure. This is especially 
important in the case of an electronic or electromechanical part 
failure, wherein the failed part will be attached to the hard copy 
and sent to Section 35^ for laboratory failure analysis. 

B. A parts analysis will be made by the Electronic Parts Engineering 
Section. 

C. The Electronic Parts Engineering Section will furnish the cognizant 
engineer with an interim or final Part Failure Analysis Report with- 
in seven calendar days after receipt of the failed part. A ditto- 
master of the final Part Failure Analysis Report will be forwarded 
to the Reliability p/FR Group for reproduction and distribution. 

Any recommendation (s ) that the Electronic Parts Engineering Section 
may have where further action is suggested will list the person 
recommended for this action. 


* This section applies to failed electronic and electromechanical 
parts. In the case of failed or faulty connectors, the same 
general, procedure should be followed, but the failed connector 
should be sent to Section 357 * 


p/FR Processing 

A. Upon receiving the pink copy from the Cognizant Section Manager, 
the Reliability P/FR Group will review the form for omissions. It 
will be the responsibility of the Reliability Project Representative 
to audit and evaluate the technical accuracy of the analysis or 
adequacy of the corrective action subject to closeout of the p/FR. 

B. The P/FR Group delivers the pink copy to the Spacecraft Systems 
Engineer for his acceptance and sign-off. The Spacecraft System 
Engineer reviews the reported description, analysis, and corrective 
action for effect on the spacecraft and its expected flight oper- 
ation, marks Block 18 of the report (critical or non-critical), 
shows concurrence by signing the pink copy, and then returns the 
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OPERATION 


pink copy to the Reliability p/FR Group where this information is 
transcribed onto the dittomaster. If the concurrence of the Space- 
craft System Engineer is not obtained, the P/FR is expedited by the 
p/FR Group with comments to the Cognizant Engineer or Section 
Manager for further action. 

C. Return of the p/FR from the Cognizant Engineer starts the sign-off 
procedure again. If agreement cannot be reached, the matter is 
referred to the Spacecraft System Manager for arbitration. 

D. The Reliability Coordinator signs the pink copy when it is com- 
pleted satisfactorily, and all addendum(s) have been received. 

The P/FR is now ’’closed.” 

Processing for Final Distribution 

A. A general distribution will be made from the "closed" P/FR ditto- 

master. The P/FR and any Parts Failure Analysis Report dittomaster s , 
as well as the pink copies of each Problem/Failure will be stored in 
the files of the Reliability P/FR Group. 

Management Visibility 

A. Monthly summaries of p/FR status will be developed, reprinted and 
distributed by Section 153 • During critical Project review periods, 
more frequent distribution will be made of these P/FR summaries. 
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•ORIGINATOR UNLESS HE IS COG. ENGR. 
TO COMPLETE SECTION I ONLY 



DISTRIBUTION 

COPIES OF ALL PROBLEM FAILURE REPORTS WILL 
BE DISTRIBUTED TO ALL CONCERNED PERSONNEL 
STANDARD DISTRIBUTIONS. BY PROJECTS. WILL 
BE ESTABLISHED BY RELIABILITY, AND 
CONCERNED EXECUTIVE PERSONNEL 


USAGE 

THIS INSTRUCTION FORM REFERS EXCLUSIVELY TO PROPER 
USAGE OF PROBLEM FAILURE REPORT FORM JPL 1 846. 

THIS FORMtJPL 1846) IS USEO TO RETYPE AN ORIGINAL 
P FR FROM FORMS JPL I 2WJPL PERSONNEL USE), OR JPL I7VB 
(VENDOR P FR). 

USE SYMBOL "N A” ON SUBJECT P'FR FORM WHENEVER 

REQUESTED INFORMATION IN ANY BLOCK IS NON-APPUCABLE. 
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(p ?m ) H PROBLEM / □ FAILURE REPORT N« 0 o^486 

□ OSE C Complex Ser. No. STC t ' OUJHOD 



Egjjll; 


SAF □ ETR Othor_ 


(HI Vendor 


8.PROBLEAVFAILURE NOTJD DURING __ ___ | INITIAL DISTRIBUTION DATE 

_ Bench Testing I I In-Procew Testing | 1 TA Te»tlng ( | FA Teetlnq XX Syefemt Testing 

Specific Environment P] Other 


9. DESCRIPTION OF PROBLEM/FAJLURE 

After returning S/C to SAP from Thermo Control Test in 25 ft. chamber. 


many Mylar chips were found lying on upper Thermo Shields. They appear 


to be comi 





10. VERIFICATION 4 ANALYSIS 

The problem 1 b caused by thermal and ultra-violet degradation of the 


ed 1/2 mil mylar material. Hie chips came from the brittle 


ltraviolet degraded portions of the sunlit solar panel squib harness 


ed with the mylar side exposed. Other sunlit areas of the harness 


and solar panel arm boots embr 


n zo protect tne myiar 


substrate from 



person completing SECTION n signoturt B . Freeman 




14. CORRECTIVE ACTION TAKEN 


All exposed sunlit cables to be wr&ppe 


e are as 


Harness Hardware 


- Cabling Integration Installation 


d Lower Ring Harness Installations; respec 

05ITI0N “ — ‘ 

D D Reodjuitod I I Stropped K1 Othor W/A 

' ~ Thl, Unit □ All Unit. B OtW All flight Unit 8 




17. review concurrence 

JV- Reliability Coordinator 8/ 

_____ Spacecraft Project Engineer 3/ 

19 , STANDARD u special distribution 

3C 

MAR-C 


T. Samuels 


THIS COPY ACCOMPANIES MATERIAL 


'27/64 

721/64 


18. classification 

H Critical 
□ Noo-Critlcol 


in 1846 OCT M 


Notes on corrective action (block 14 ): 

1. Tedlar has materially better ultraviolet resistance than mylar. 

2. Pigmented film helps prevent wrapping with the wrong side out. 

3. 2-mil Tedlar will stretch less in wrapping than %-mil mylar. 
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"The aeronautical and space activities of the United States shall he 
conducted so as to contribute ... to the expansion of human knowl- 
edge of phenomena in the atmosphere and spate. The Administration 
shall provide for the widest practicable and appropriate dissemination 
of information concerning its activities and the results thereof ” 

— National Aeronautics and Space Act of 1958 


NASA SCIENTIFIC AND TECHNICAL PUBLICATIONS 


TECHNICAL REPORTS: Scientific and technical information considered 
important, complete, and a lasting contribution to existing knowledge. 

TECHNICAL NOTES: Information less broad in scope but nevertheless of 
importance as a contribution to existing knowledge. 

TECHNICAL MEMORANDUMS: Information receiving limited distribu- 
tion because of preliminary data, security classification, or other reasons. 

CONTRACTOR REPORTS: Scientific and technical information generated 

under a NASA contract or grant and considered an important contribution to 
existing knowledge. 

TECHNICAL TRANSLATIONS: Information published in a foreign 
language considered to merit NASA distribution in English. 

SPECIAL PUBLICATIONS: Information derived from or of value to NASA 
activities. Publications include conference proceedings, monographs, data 
compilations, handbooks, sourcebooks, and special bibliographies. 

TECHNOLOGY UTILIZATION PUBLICATIONS: Information on tech- 

nology used by NASA that may be of particular interest in commercial and other 
non-aerospace applications. Publications include Tech Briefs, Technology 
Utilization Reports and Notes, and Technology Surveys. 


Details on the availability of these publications may be obtained from: 


SCIENTIFIC AND TECHNICAL INFORMATION DIVISION 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 

Washington, D.C. 20546 


